The Four Hundred

OS/400 is not a one-trick pony. It was designed to be an operating system that can do many things at once, in such a way that different applications in contention for resources do not adversely affect one another. While IBM has run database, Java, e-mail, and other single-workload benchmarks on the OS/400 platform for 15 years, to show how it measures up to other machines running a single workload, no benchmark has been developed to show the multiple workload capabilities of OS/400. Until now.

IBM announced a new test last week that it has run on the iSeries, called the three-in-one benchmark, which illustrates this capability of the OS/400 platform. While mainframes have similar workload managers that do a good job of isolating workloads on a single machine so they can share that machine without affecting response times, Windows and Linux servers do not have the same level of multitasking sophistication out of the box. Companies have to use virtual machine partitioning software, like that available from VMware, SWsoft , and Microsoft (which acquired a company called Connectix and has Windows partitioning software in beta right now) on Windows or Linux platforms to isolate workloads like OS/400 does right out of the box with its subsystem capabilities and without any tuning or tweaking by system administrators. Even the much-improved Unix workload managers in AIX, Solaris, and HP-UX usually have to resort to logical partitioning to give the kind of workload isolation that OS/400 does by default.

The three-in-one benchmark combines an ERP-like benchmark called the Java Business Object benchmark (jBOB) with a set of Lotus groupware software and a Web storefront, all running on the same machine. The jBOB test is, sources at IBM say, a Java implementation of the same code that is used in the TPC-C online transaction processing test. The TPC-C test simulates order processing, inventory, and other operations associated with running a distribution warehouse. IBM warns that this is not a version of the TPC-C test that is sanctioned by the Transaction Processing Performance Council. Incidentally, the SPECJBB2000 test is the same Java code, minus its database and I/O processing; because this test is only designed to stress-test an individual JVM, these other elements are removed. Obviously, either TPC or the Standard Performance Evaluation Corp., which administers the various SPEC tests, could implement the jBOB benchmark, which is several years old. But they have decided not to, and this is completely stupid. The other workload in the three is based heavily on the NotesBench benchmark tests that IBM has used for years to demonstrate the performance of Notes and Domino middleware. The third component is called Trade2, and it is a simulated workload built using Java servlets, JavaServer Pages, and Enterprise JavaBeans to simulate the processing of an online store.

Many of us have been clamoring for IBM to develop a benchmark like the three-in-one test for the same reason why Big Blue has finally done it. No Windows or Linux server can run multiple applications with consistent response times for those applications on a single instance of the operating system. This means a Windows or Linux server is often running one job at 10 percent, 15 percent, or maybe 20 percent of total CPU capacity most of the time, and that as companies add workloads--ERP, e-mail, and Web serving, for example--they have to add machines. While Wintel and Lintel iron is inexpensive, it isn't, when you examine the cost per clocks actually used in a machine. A machine that can only use a quarter of its capacity to do a single workload, and cannot use the other three quarters of its capacity to do something else, is four times as expensive as you think it is. I went into this in my iDeal iSeries stories at the end of last year, and showed that if IBM demonstrated the multiple application capabilities of the iSeries, it could demonstrate that an iSeries acquisition made more sense than buying multiple Wintel or Lintel servers.

IBM has, unfortunately, stopped short of making any direct comparisons between the iSeries running the three-in-one benchmark and other platforms doing it. The thing to remember, though, is that virtual machine partitioning software is not free on either Windows or Linux. VMware GSX Server costs $2,500 on a machine with one or two processors and $5,000 on a machine with three or four processors. ESX Server, which more fully isolates virtual machine partitions, costs $3,000 on a two-way machine and $10,000 on an eight-way machine. In many cases, products like GSX Server and ESX Server only make sense for bigger shops that have lots of servers to consolidate onto one machine. You can put dozens of virtual machines on a single physical server, each with its own operating system, with either GSX Server or ESX Server. Spreading that VMware licensing cost over many virtual machines makes it negligible. But not so, if you just want to carve up a single machine into a few partitions. Nonetheless, virtual machine partitioning allows companies to get CPU usage on Wintel and Lintel iron up into the same ranges as an iSeries does right out of the box, for multiple workloads, and it is clearly better than server sprawl.

Here's what IBM did on the three-in-one benchmark test. First, it grabbed a two-way iSeries Model 810 off the production line. This machine uses two 540 MHz S-Star PowerPC processors and is rated at 750 CPWs of processing power. The exact configuration of the machine is unclear, but it was running OS/400 V5R2 with Domino R6 and WebSphere Express. IBM first ran all three workloads independently on the Model 810. The jBOB online transaction processing test used 17 percent of the aggregate processing capacity of those two S-Star processors and was able to handle 24.4 Java transactions per second (JTPS) supporting 500 simulated users. Each S-Star processor had its own JVM to run the jBOB test. On the collaboration part of the test, 500 users simulated the use of Domino Web Access and Lotus Instant Messaging. This part of the workload ate up 25.5 percent of the processing power of the Model 810, and response times for e-mail and Instant Messenger were 0.218 seconds and 0.197 seconds respectively. In the Trade2 Web-serving environment running alone on the iSeries, 1,125 simulated online shoppers were supported using 23 percent of CPU and yielding 61 transactions per second (TPS) with an average response time of 0.202 seconds.

Now here's the good bit. When all three of these applications were run simultaneously on the exact same Model 810, supporting the same number of users, the CPU utilizations and average response times for the three workloads remained nearly the same as when they were running alone on the machine. The total number of users--500 jBOB, 500 Web e-mail, 500 Instant Messenger, and 1,125 Web store--running on the three applications consumed 70.6 percent of total processing capacity with the same throughput and response times. The differences were not statistically significant.

In the course of the test, IBM jacked up the number of Web store users to 2,850 users, and this pushed overall CPU use to 96.8 percent. Still, the average response time and throughput of jBOB and Instant Messenger users stayed the same. Web e-mail users saw response times increase to 0.654 seconds, and Web-serving response times increased to 0.509 seconds as the machine started to run out of gas. But the point is, the Model 810 dealt with a substantial spike in online store processing without adversely affecting the other workloads on the machine. IBM also generated intermittent spikes in demand on another variant of the three-in-one test by adding Lotus Web Conferencing users to the mix and having them start and hold 50 meetings per hour. While this pushed up CPU use on the Lotus stack, it did not affect the response times or throughput of the jBOB OLTP or Trade2 Web-serving benchmarks in any appreciable way.

You can find out a lot more about the three-in-one benchmark test on IBM's Web site. While IBM has only run the test on the single Model 810, the workload estimator on the iSeries site (which is linked to on the above page) can simulate the benchmark on any iSeries machine. You can also change the parameters in the benchmark to more closely match your own workloads.

Jelan Heidelberg, business development manager for Lotus at IBM's Systems Group, says that IBM has only done this three-in-one test as a one-off benchmark on the Model 810 because it wanted to get the message into the market quickly and to give something that resellers can use to sell the iSeries against alternative platforms. If the iSeries channel reacts positively to this test and it helps to sell iSeries iron, IBM may champion the test further and may even run direct comparisons with other platforms. It would be even more helpful if IBM championed an open benchmark administered by a third party that included audited pricing information, much as the TPC benchmarks do. The SPEC benchmarks do not including pricing information, which is their real drawback.