Sutter Health Finds a Cure for Planned Downtime

by Alex Woodie

Sutter Solano Medical Center uses the Midseries 4 application suite, originally developed in the 1980s by Intermountain Health Care, and now supported by Siemens AG. The hospital uses Midseries 4 to handle various financial aspects of its daily operations, such as admitting new patients and accounts receivable.

Like other hospitals, Sutter Solano is mandated by state laws to ensure that downtime is kept to a minimum and that the hospital can safely perform its duties. Ideally, regular maintenance would not require the hospital to shut down its AS/400. However, for some tasks, such as purging old files and reorganizing the existing ones to reclaim unused disk space, it almost always does.

Whenever Sutter Solano performed a file reorganization on its single-processor AS/400 Model 720, it normally required taking the Midseries 4 application offline for 8 to 10 hours, says Jim Barringham, special projects manager at Sutter Solano. During this time, none of the hospital's nursing or administrative staff was able to access the application. Since this length of downtime was unacceptable during the day, the hospital minimized the impact by breaking the file reorganization into multiple sessions, which Barringham ran during off-hours of the week.

The situation was compounded by the fact that the disk capacity in the hospital's AS/400 Model 720 was above 90 percent. IBM recommends that AS/400 shops keep their disk utilization below 80 percent to maximize application response time. Barringham did not have a good set of choices: reorganize the files and create downtime or spend a considerable sum to buy more disk from IBM. There had to be a better way.

A New Way to Do File Reorgs

About a year ago, Barringham heard of a new software utility that would allow him to keep his AS/400 online while reorganizing the files. The utility, Reorganize While Active, was developed by a young Salt Lake City company called iTera.

Barringham was familiar and felt comfortable with the people at iTera, which was founded by Dan NeVille and Kent McDonald in 1997. Barringham successfully used iTera's Y2K conversion tools, which were the precursors to iTera's current lineup of continuous availability solutions. Barringham's confidence in iTera and its developers and technology was further bolstered by the fact that NeVille and McDonald previously ran a consulting business that specialized in the Midseries 4 application.

But Barringham was still skeptical about the Reorganize While Active utility. Would it really work? Could it handle the customized features he built into his employer's files? To allay those fears, iTera referred him to the University of Las Vegas, Nevada, Medical Center, which was also using the utility in a Midseries 4 environment.

Barringham checked the reference. The hospital said it worked, so Barringham figured he would at least give it a shot. "Basically, I decided that if this dang thing works, it's a hell of a deal," he said.

How It Works

Reorganizing files is a regular part of system maintenance on an AS/400--or on any computer, for that matter. The process is crucial to reclaim disk space that is occupied by deleted files. IBM includes a command called Reorganize Physical File Members (RGZPFM) in OS/400 that just about every AS/400 and iSeries shop is (or should be) familiar with. But the downside of using RGZPFM is that it requires the application to be brought offline while the reorganization takes place. It also consumes so much of the AS/400 or iSeries processing power that running any other applications is practically impossible.

Reorganize While Active allows users to access the application and its files while a file reorganization is taking place. It consumes its share of batch resources, but at least users have access to the application and its underlying database during the reorganization.

The utility works by creating a duplicate set of all the files involved in the reorganization. As the file records are being copied into a duplicate library, it separates those tagged for deletion, thereby regaining the lost disk space.

But if that's all it did, Reorganize While Active wouldn't be much more useful than RGZPFM. To allow users to continue to use the files while they're being reorganized, the utility keeps the files synchronized by matching the sets using relative record numbers. As changes are made to the original files in the production environment, the utility simultaneously makes the same changes to the files in the reorganization environment, a process that continues until the reorganization is completed with one final file swap.

A drawback of the current version of Reorganize While Active is that it requires additional disk space to create the duplicate sets of files, which is not a good thing if you're nearing disk capacity, as was Sutter Solano Medical Center. However, iTera is currently developing a new version of Reorganize While Active that doesn't use the duplicate-file process. (Additional information on this new version of Reorganize While Active will be detailed in future Guild Companies newsletters.)

Reorganizing in Action

Despite the encouraging reference from the UNLV Medical Center and his positive past dealings with iTera personnel--as well as the fact that his AS/400 was screaming for more open DASD--Barringham was still skeptical about the utility. Erring on the side of prudence, he decided to slowly roll out the utility and closely monitor its performance along the way.

Barringham started running Reorganize While Active Version 2.1.1 on some of the smaller, less significant files at the hospital during a week in late November 2001. If the software was going to crash his AS/400, he wasn't about to lose the file with five million records in it. As the week went by, he gradually started running the software against bigger and more complicated files, until he built up a comfort level with the product.

"The last file that I did was one that had triggers on it," he said. "I didn't want this file being rebuilt without my triggers. I was worried whether they had thought about that [supporting the triggers]. You never know when this code was done, and triggers are a relatively new thing. It was new to me."

To Barringham's surprise, the utility worked as promised, triggers and all.

"That was the most amazing thing, that it actually worked," he said. "It was actually almost too easy. I kind of out-thought it."

2002 Update

As a result of the first reorganization completed with the utility, Barringham reclaimed about 20 percent of his disk space, putting him in the low-70-percent-DASD-utilization range. In terms of return on investment, Barringham said he recouped the hospital's investment with that first file reorganization.

Barringham has used the utility at least one other time, and continues to use it on an as- needed basis. All in all, Barringham estimates that Reorganization While Active has saved the hospital a total of 10 to 12 hours of system downtime.

Since then, however, downtime as a result of system maintenance has become an even bigger issue at Sutter Solano. In January 2002, the hospital implemented a new AS/400- based application that allows doctors to view the results of lab tests and radiological exams. Now, if the AS/400 becomes inaccessible, it's not merely an inconvenience to the doctors and the clerical workers; it could also have an effect on the care that the hospital's patients receive.

"Downtime is definitely more critical now" compared to when he originally purchased" the software, Barringham said. "If the interfaces are down, the doctors can't look at their results from radiology or the lab."

In terms of customer support, Barringham said he didn't deal much with iTera. He did contact the company about a change in documentation and was quickly put in touch with a person who handled his questions.

Reorganize While Active eventually overcame Barringham's initial skepticism and proved itself to be a worthy member of his toolbox. "Bottom line, the dang thing worked," he said.