Centerfield Puts Mutex Waits on Notice

by Alex Woodie

The iSeries performance experts at Centerfield Technology are developing a new utility that will help system administrators keep their applications functioning properly by detecting iSeries jobs that have entered mutual exclusion, or mutex, wait states. Mutex conditions, which are commonly used in Java and Unix applications, are increasingly being found in ported OS/400 applications. However, because of the way they work, Mutex waits have the possibility of hurting other jobs, even those that are functioning properly.

During the normal course of business, iSeries administrators should expect to watch their database applications go through various states. As part of their OS/400 application needs exclusive access to a certain object or record to complete its cycle, the job will enter the traditional object and record lock state (LCKW), or even the semaphore wait (SEMW) display (which is not as common as LCKW), until it becomes free.

Administrators at iSeries shops running Unix and Java applications are beginning to see other kinds of wait states on their work-with-active-job (WRKACTJOB) displays, according to Centerfield Technology, a Rochester, Minnesota, software company that specializes in iSeries performance problems. Specifically, iSeries shops running applications such as WebSphere, or Oracle's EnterpriseOne (formerly J.D. Edwards OneWorld) are beginning to see mutex waits (MTXW).

Mutex waits are not to be feared. They are commonly used in Java programming to synchronize resources in multi-threaded applications, and are analogous to LCKW or even SEMW conditions, says Elvis Budimlic, a development manager with Centerfield. "Usually it's a good thing," he says. "There is a reason for mutex waits. But there's no reason to wait for an hour."

The problem with mutexes is that one faulty job or thread can cause other jobs that are waiting for the mutex to erroneously enter the wait state. "Every once in a while, due to a bad program or operator error, these things don't get released. Then your iSeries administrator doesn't know what to do," Budimlic says.

Centerfield's answer to this problem is mutex/DETECTOR. Instead of allowing an application experiencing a mutex wait condition to adversely affect your other jobs, administrators will be able to tell which jobs are holding the mutex and which jobs are waiting for the mutex to be released before they can continue.

Armed with this information, administrators can decide whether to allow the mutex wait to continue, or to kill the offending job and let the surrounding jobs continue about their way. The mutex/DETECTOR also keeps logs to help developers and ISVs trace the mutexes, so they can fix the problem.

Centerfield initiated development of mutex/DETECTOR after one of its customers, an EnterpriseOne user, basically had its ERP application brought to a standstill as a result of a mutex hold that didn't go away, Budimlic says.

iSeries administrators should become familiar with mutex holds as Java and ported Unix applications become more popular on the iSeries, Budimlic says. "They're going to be using mutexes because that's what they use for other platforms. It's really becoming a problem," he says. "This will become much more common. You have a lot of threads in these Java programs. You need to have some sort of control in them to synchronize actions."

Centerfield is currently developing mutex/DETECTOR, and says it will become available toward the end of the third quarter and the beginning of the fourth quarter. It will be bundled with the next release of insure/MONITOR and insure/MONITOR for OneWorld iSeries diagnostic tools. Pricing for the insure/MONITOR products ranges from $7,500 to $16,000. For more information, visit www.centerfieldtechnology.com.