Midrange Stuff, OS/400 Edition

Try to imagine a planned downtime stretching on for hours. The IT staff works through the night so critical applications can be backed up and running at the start of the next business day. Chances are that some of you don't have to use your imaginations. It's a very real part of your jobs. If you could substantially reduce that downtime, how good would that be? This story describes how it was done in the financial industry, but the example could apply to other markets as well.

Jack Henry & Associates is one of the best-known names in the banking industry. The Monett, Missouri, company develops and acquires banking application software systems and markets them throughout the United States. As part of its services package, the company offers data conversion, software installation for the implementation of its systems, and outsourcing options.

Jack Henry customers let the company know about concerns they had about the amount of downtime associated with software upgrades, file reorganization, and database conversions. It was infringing on the critical needs of modern financial institutions. The company responded to its user base by making the downtime issue a priority for the in-house software development team.

In previous releases of the Silverlake System, Jack Henry's core OS/400-based processing application for financial institutions, the upgrades were sent to customers on tape. When the banking firms undertook an upgrade, all users had to be off the system, all ancillary products had to be offline, and customers could not access important tools like ATM services. This was necessary for file conversion. Depending on the size of the iSeries or AS/400 and the amount of DASD utilized, the conversion could take a minimum of three hours or as much as 19 hours.

Customers chose to handle this process in different ways. But the trick that each had to deal with was handling their normal end-of-day procedures--which typically takes between three and five hours--as well as managing the upgrade. The upgrade was always done at night, when demands on the system were at their lowest.

Some IT staffs had chosen the parallel approach that meant processing as normal, then doing a process backup of data libraries and several other libraries used by the Silverlake system. They followed that step by restoring all the libraries back onto the AS/400, then beginning the conversion. In this scenario, the same data was being processed twice to allow comparison. Because there were two independent databases involved, it was possible for errors to result.

Those who handled their normal end-of-day processes either before or after the upgrade was completed often did not have time before the start of the next business day to test the new interfaces, customizations, and modifications. The suspense of discovering whether the upgrade went without a hitch was postponed until the following night.

The latest release of Silverlake eliminates the parallel and the "before" and "after" scenarios. Mike Gollhofer, assistant manager of the Silverlake release department, says the main ingredient in this accomplishment is the capability to convert the large files and information during the day and journaling the changes, which gives a huge window before doing the normal end-of-day requirements. The tool that allows this to happen is TurnOver PDQ from SoftLanding Systems.

TurnOver PDQ is the key to overcoming several obstacles that concerned Jack Henry customers. It allows the simultaneous revision of data files, as well as upgrading and reorganizing data files with minimal interruption in service. The emphasis is on minimal interruption.

PDQ uses change management technology, which is the forte of SoftLanding, and also incorporates high availability technology, because PDQ is designed in combination with Lakeview Technology's MIMIX/Promoter, a product that allows database file reorganization and restructuring in a live production environment. With TurnOver PDQ, Jack Henry users were able to maintain read/write access to databases while upgrades took place.

By using PDQ, customers were able to create an up-front testing scenario two weeks before the live date. Because of the improvements, information was converted to the new release while simultaneously allowing testing and inputting of transactions. It allowed customers to run a subset, an independent scenario of live production information, which was converted to the new release. "When it goes live," Jack Henry's Gollhofer says, "there are no worries about restoring data libraries or trying to do two parallel processes. Our customers are no longer copying files from one data set to another, because they are now using the same database."

Gollhofer says many Jack Henry customers were already using the SoftLanding TurnOver products, which made the introduction of SoftLanding's PDQ "a seamless integration." "This was a piece on our side that provided the high availability during maintenance functions," Gollhofer says. "You can't get around the maintenance functions. One of the most important [functions] is the reorganization of files, and a big benefit there is the saved DASD and the money saved when compared to buying DASD and the hours of IT staff time required when doing these things."

DASD utilization is a big part of this equation. "Once you hit the 80 percent, or so, threshold, everything really slows down," Gollhofer notes. When a machine is heavily taxed, performance suffers, and Gollhofer heard from customers who were experiencing system degradation while converting information.

He liked the idea of incorporating PDQ because it allows the creation of a separate subsystem for the PDQ product to run the conversion jobs. He points out that this is done to avoid taxing the interactive or batch subsystems.

Jack Henry customers are running Silverlake on a variety of AS/400 and iSeries boxes, everything from Model 500s to Model 830s. Processor size is certainly a factor in performing these banking tasks, so Gollhofer keeps records on the downtime that customers experience as they go through the release. He showed these findings, based on a user survey, at the Jack Henry user conference in October.

Here are the figures based solely on the processor size of the AS/400 or iSeries:

Gollhofer notes that direct conclusions from these statistics should not be made, because factors in addition to processor size are relevant. As an example, he says a billion-dollar bank with a relatively large number of files to convert may use a Model 720. Another institution with a larger machine may have fewer assets and files to convert. Some Jack Henry customers have gigantic machines for growth purposes, yet those institutions may not necessarily have those boxes working anywhere near capacity. A bigger machine, at a bank with smaller assets, means less downtime, even without PDQ.

Other factors that Gollhofer takes into consideration when plotting the level of success of the latest Silverlake release include DASD utilization and the number of institutions (multi-bank facilities) that are involved in the upgrade, file reorganization, and subsequent down time.

One of the biggest questions at the Jack Henry user group meeting, Gollhofer says, was whether PDQ would be required for Silverlake Release 13 and beyond. According to Gollhofer, the answer is no. "We would like all our customers to be on the product, but we understand the smaller institutions may not believe they get benefit. We will not force them.

"When it was first proposed to the Jack Henry customer base that they should buy PDQ, there was a lot of resistance. Now it has turned around, and people are actually supporting a plan that would require everyone to have PDQ." The reasoning was so that the Silverlake development team could then focus all development in one area. "We are still developing and adding functionality and features that utilize PDQ, but I do also have to develop for those that won't have PDQ capabilities," he says.

The latest edition of Jack Henry's Silverlake, Release 13, is due later this month.