Exercises in Simplicity

iSeries servers don’t require a lot of muss and fuss. They typically just keep running, day after day, which can be bad if you’re in the support business (remember the Maytag Man?) or in the server business (IBM secretly wishes your trusty old AS/400 would die so you’d finally upgrade to a new iSeries). While the OS/400 ecosystem is, in a way, the victim of its own success, customers such as the USDA and Sun National Bank benefit by running lean and mean.

With 2,441 AS/400s in use, the USDA’s Farm Service Agency has to be one of the biggest OS/400 shops in the country. Each FSA county office uses a baby AS/400 Model 170 to perform its key functions, such as signing up farmers for government assistance, tracking and paying farm subsidies, publishing newsletters, and other common office tasks.

The FSA centralizes as much of the management of those AS/400s as possible from its headquarters in Kansas City, Missouri. This is where the COBOL code is maintained, where OS/400 PTFs and application updates are distributed, and where Level One technical support is provided. With more than 2,400 virtually identical 170s installed in each county office in all 50 states (and Puerto Rico and Guam), the Kansas City office has a pretty good idea of what those 170s should look like, even from afar.

But when it comes to taking care of life’s little hardware problems, you need feet on the ground. In 2003 the USDA decided to outsource the maintenance of those 2,441 AS/400s, as well as 7,400 IBM printers, to an outside firm called Communication Technologies, or COMTek.

From its headquarters near Washington, D.C., COMTek has become a trusted supplier of technology know-how to the federal government, including the State and Defense departments, as well as various corporate entities. Last week, COMTek announced that the USDA has extended the original FSA maintenance contract through the fall of 2007 for $9 million.

Support Circles

So what does the FSA contract have to do with the AS/400’s legendary reliability? Consider this: COMTek manages to service those 2,441 AS/400s and 7,400 printers located in the farthest reaches of the United States using a staff of only 35 people. If you run the numbers, this means each COMTek tech support person is responsible for maintaining an average of 70 AS/400 servers and more than 210 printers.

Mark Brandon, the FSA contract manager, has also run a lot of numbers to figure out where exactly those 35 tech support people need to be located to deliver the guaranteed four-hour response time for each office. “If you take a map of the country, and draw a four-hour circle around each FSA office, figuring an average speed of 60 mph, you only need five or six circles to go from one end of the country to the other,” he says. “It also helps that 70 percent of them are east of the Mississippi. And my techs are willing to do a lot of driving.”

Brandon can fine tune where those circles need to be thanks to two years worth of hard data on the failure rates of those Model 170s and IBM printers, and his 96 percent success rate for delivering service within eight hours. About 85 percent of the service calls are for the printers, which is largely due to the fact that the government requires COMTek to install refurbished toners. Some states with a lot of farms, such as Indiana, also have higher call rates. There is also a heavy volume of calls out of the Carolinas, “for obvious reasons,” he says.

“Those particular [AS/400] servers cause me about two service calls a day. That’s their realized reliability,” Brandon says. “It’s actually lower reliability than what I expected. Over the life of contract, every one is going to fail. But I’m prepared to deal with that.” Brandon has seen just about every type of failure over the life of the contract, and has found that tape drives and hard drives account for most of the problems. Thanks to built-in redundancy, these problems don’t often result in downtime.

What would COMTek’s support circles look like if the FSA used your industry standard PC server instead of the rugged and dependable AS/400? Brandon played along and waged a wild guess (and it’s just a guess, he stressed). “If they were PC servers, I would expect a higher failure rate, and therefore we would need smaller circles,” he says. “I would guess that those PC servers would exhibit a 75 percent higher failure rate, and it would probably require a 75 percent increase in staff.”

Sun National Stays Lean

While the AS/400’s low failure rate enables the FSA to cost-effectively outsource maintenance of its widely distributed computing network, another OS/400 shop, Sun National Bank of Vineland, New Jersey, is benefiting from the iSeries reliability in a different way: by centralizing and consolidating its critical data onto one platform.

IBM today announced that Sun National Bank has upgraded to a new eServer i5 Model 550 to run its $3 billion lending business. The new i5, like the iSeries Model 820 before it, forms the backbone of this mid-sized community bank, while requiring just a handful of IT staff to run it.

Sun National’s implementation of Kirchman‘s OS/400-based core banking software forms the foundation for every customer interaction. Whether transactions are occurring through ATM machines, over the Internet, via telephone, or in person at one of Sun National’s 73 branch locations, those transactions are hitting the iSeries server. Some of Sun’s applications, including the teller and telephone banking apps, run on a network of 73 xSeries servers that are “loosely coupled” to the iSeries database through MQ Series.

Keeping all account information centralized on the iSeries is important, as it provides “a single version of the truth,” says Lou Pellicori, Sun National’s executive vice president and chief information officer. “We have been growing substantially in the last year,” he says. “We bought it to position ourselves for continued growth.”

Positioned for Growth

Pellicori outlined four key aspects of the iSeries that made it his continued choice for Sun National Bank, including the capability to run multiple operating systems, total cost of ownership (TCO), security, and reliability and recoverability. “Technology is an uncertain journey,” Pellicori says. “With the i5, we’ll be able to run on other operating systems, if needed. We can run Linux, AIX, and Windows, if we needed to run those other infrastructures.”

Pellicori says the iSeries has a better TCO than other platforms he has experience with, including Windows and mainframe. “This one [iSeries] seems to have a lower TCO from a staffing and an overall perspective,” he says, adding that a staff of only several people is able to handle all operation and maintenance activities across three shifts. “It’s a little bit like lights out, which I like.”

To gauge security, Pellicori looked at the rate of virus and system breach vulnerabilities on Linux and Windows systems, and the average time it took system vendors to patch those holes. “These percentages are a little daunting” on Linux and Windows, he says. “When it comes to AS/400 you’re not looking at viruses and outside penetration.”

Finally, Pellicori says the iSeries’ recovery facilities, including the capability to take snapshots of OS/400 configuration and user libraries, and the iSeries RAID-protected disks, makes getting back up and running a cinch. “That environment is a pretty simple environment to recover,” he says, noting that, in the three years running the single-processor iSeries Model 820, the bank never experienced any downtime. “If a disk failed, IBM came, slid in the new disk, and we would continue to run,” he says.

Sun National is also positioned to further simplify its computing environment, specifically those 73 xSeries servers used to run the teller and phone systems. By consolidating those Windows boxes and connecting them to the iSeries through the integrated xSeries adapters, Pellicori would be able to utilize his iSeries disk, and simplify his nightly back-up at the same time. “It would simplify business continuity, we’re just not there yet,” he says.