Midrange Stuff, OS/400 Edition

Shell Canada Ltd. is one of the largest iSeries shops in North America. The company operates a lights-out data center in Calgary, Alberta, that consists of four large iSeries systems running J.D. Edwards applications, among others. As a fully automated data center, Shell Canada is very attuned to the efficiency of its infrastructure, and keeps an especially close eye on its 7 TB of iSeries DASD. However, when the IT staff started noticing spikes in DASD usage, they rightly became concerned.

The company, a subsidiary of Royal Dutch Shell, is conservative in its approach to maintaining a water-tight systems infrastructure, even by conservative AS/400 standards. The company had configured its iSeries servers to alert the IT staff, via the paging module of Bytware's MessengerPlus, whenever disk utilization exceeded 70 percent. Typically, however, disk utilization resided quite comfortably around the 50 percent mark, said Kirk Chalk, a Shell Canada senior staff systems analyst.

But sometimes, especially around the end of the month, when people were running lots of reports against the OneWorld and WorldSoftware databases (Shell Canada runs them in co-existence mode), the disk utilization on some of the iSeries would get out of hand. For example, on one 12-way iSeries Model 840 server equipped with almost 3 TB of DASD, there was once only about 180 GB of unused space, a 96-percent utilization figure. This is not a good situation to be in, even if you have oodles of storage, as Shell Canada does.

It didn't take a lot of time for Chalk to figure out why this was happening. The way that DB2 is architected, Chalk explains, the database will perform a number of table joins and build indexes to satisfy the users' requests for information. Some of these requests can result in large, temporary objects being built and stored in the database, some of them as large 5 GB to 30 GB for a database as big as Shell Canada's. Also, the proliferation of nontraditional applications in Shell Canada's shop--applications such as WebSphere, which Shell is introducing to provide a Web interface for their OneWorld implementation--was increasing the utilization of OS/400' IFS for production storage. The least offensive culprit in the escalation of DASD usage at Shell Canada were outfiles created by batch programs, he said.

Knowing where the problem is generally coming from is one thing. But having the information to actually tackle the problem in an efficient manner was entirely another.

"We had a number of incidents occurring where we'd get tremendous spikes," Chalk said. "We're fairly sophisticated in our storage management strategy, but we didn't have a way to get a handle on dynamic spikes for database queries."

OS/400 shops are fairly limited in the solutions available that address this specific problem. The operating system can be configured to sound an alert when a certain percentage of disk utilization is reached in an auxiliary storage pool. But that doesn't provide the operators with the detailed information they need to act on it and prevent damage.

OS/400 also ships with a free facility called Performance Explorer, or PEX, that can provide a more in-depth view of how the disk is being used and who is responsible for using it. However, PEX can be difficult to use effectively when time is limited, because it generates a mass of information that the operator must sift through in order to find the offending job that is gobbling up the DASD and the user who created it. When the iSeries is teetering under an extremely high disk utilization load, you don't want to be dallying around. You want to get right to the problem.

The lack of a good utility in this category concerned Chalk, who has experience with other large systems such as IBM mainframes, which Chalk helped Shell Canada migrate off of several years ago, in favor of AS/400s. "There's no facility on the AS/400 that gives you a snapshot of the delta in storage over a defined interval," he said. "'Where did I lose 100 GBs in the last hour?' There's no easy way to find out on these machines."

Shell Canada had a good working relationship with a utilities vendor named Centerfield Technology, which is based in the iSeries' home town of Rochester, Minnesota. Shell Canada had had good experience with CTI's insure/TOOLS performance management tools, and about six months ago Chalk decided to see if Centerfield had an answer to the Shell's problem.

As it turned out, CTI didn't have a solution. At least not then. But CTI had another client, Tiffany & Co., that was experiencing a similar problem. That's when CTI decided to develop disk/HUNTER, with the majority of input from Chalk and Shell Canada.

CTI announced the first release of disk/HUNTER in January. The utility basically takes the disk utilization results returned by the PEX tool and compiles them into a format that's easier to read and use. With disk/HUNTER running, systems administrators can quickly find out which jobs are consuming disk, what objects are associated with those jobs, and who created the job--all the things that an administrator needs to know when he has to manually go in and stop the process, and do it fast. A license to use the utility ranges from $1,750 to $6,750, depending on the tier of the user.

Shell Canada implemented disk/HUNTER about a month ago, and set it up so that, if disk/HUNTER detected an anomalous disk situation, it would send a message to the MessengerPlus system, which would then send a page to the Shell Canada IT staffer on call. The company has experienced one month-end processing spurt since it was installed, but, luckily, there were no critical disk usage situations where disk/HUNTER could play hero.

While there have been no critical situations yet, Chalk has taken the opportunity to put disk/HUNTER to work in a little proactive clean-up of the systems storage. "There are reactive aspects, and there are also proactive aspects. You can get this thing to generate reports," he said. "It was very revealing to see where all the temporary storage is being allocated."

Some applications were leaving trails and logs that the administrators didn't know about, and probably wouldn't have without disk/HUNTER. "We set up procedures to clean up and remove them," Chalk said. "We have great hopes for disk/HUNTER in the future."

As Shell Canada begins to move away from an RPG-centric approach to its applications and toward a DB2-centric approach, it will likely have more disk usage spikes to deal with. Chalk says that the way in which RPG allocated disk resources is much more predictable than DB2. But now that he's armed with the disk/HUNTER, he's not so worried.

"If you're within a few megabytes from losing the box, it's really a terrible situation," Chalk said. "There are a few ways to hunt and peck to find it out, but now we can nail them in their tracks."