The i5 Shows Linear Scalability on SAP Benchmark

by Timothy Prickett Morgan

IBM has shied away from running SAP's Sales and Distribution (SD) transaction processing benchmark on the i5 servers, and has instead tried to stake a claim over on its Business Information Warehouse (BW) benchmark, which measures the data warehousing performance of the machine running SAP's software for performing that function. While results on the SD test would be nice for comparison shopping, a new result on the i5 servers does a different and equally useful thing: it demonstrates that the i5, like the p5, is able to scale linearly on real applications.

The SAP BW test, which examines the performance of a server in loading two sets of data into the warehouse (measured in rows per hour) and then running queries (which are measured in navigation steps per hour) against that data. Not too many recent machines have been tested using this benchmark, which limits its utility. But it has nonetheless been useful for iSeries shops for reasons I will explain below.

IBM's latest i5 to be tested on the BW benchmark test is an i5 570 with eight 1.65 GHz Power5 processor cores and 16 GB of main memory. This is basically twice the iron that IBM tested in December 2004 and then again in March 2005 after making tweaks to i5/OS that significantly boosted the performance of database queries on workloads like SAP's BW application. Those tweaks were also used on the i5 570 tested, and that eight-core server presumably benefited as much when it came to query performance because of those tweaks. This machine loaded up 79.4 million rows in the first phase, then 18.4 million rows in the second phase; then, in the query phase, it was able to process 179,208 navigation steps with an average processor utilization of 99 percent.

In the initial i5 550 server test, IBM used SAP R/3 4.70 application code and the 3.0 B edition of Business Warehouse. This server was able to sustain an average throughput of 34.3 million rows per hour in the first load, 11.2 million rows per hour in the second load, and 78,948 query navigation steps per hour in the query phase of the test. During the latter part of the test, the machine was able to sustain 96 percent CPU utilization. This box was configured with four 1.65 GHz Power5 processor cores, 8 GB of main memory, and i5/OS V5R3 with its integrated DB2/400 database.

In March of this year, IBM configured an i5 550 with i5/OS V5R3 and the new SQL Query Engine and materialized query tables (MQT) enhancement that it made available as a PTF patch. This i5 550 was able to load 34.4 million rows per hour in the first load and 11.4 million rows per hour in the second load--hardly no change at all. But with the SQL enhancements, IBM was able to get the CPU utilization during the query part of the data warehousing test from 96 percent up to 99 percent and was also crank through 90,324 navigation steps--an increase of 14 percent without any change in hardware and facilitated by a software patch. The latter i5 550 configuration as well as the new i5 570 configuration were tested using the R/3 Netweaver '04 release of the ERP suite and the 3.5 release of the Business Warehouse data warehousing software from SAP. There is no way to be sure how much of the performance to changes in the code inside DB2/400, SAP R/3 itself, the Business Warehouse, or NetWeaver. But it is probably safe to say that a lot of the performance gains in moving from one i5 550 to the other and about 15 percent of the performance in the i5 570 was due to the SQL enhancements in DB2/400.

So, there are two interesting bits of data that you can derive from these tests. First and foremost: those SQL patches for i5/OS are something customers should investigate and put on their systems. The fact that the scalability of moving from a four-core box to an eight-core box is 179,208 divided by 90,324, or a factor of 1.984, is a testament to the design of the i5 servers. A four-core i5 550 is rated at 12,000 CPWs of raw processing performance using IBM's the Commercial Processing Workload that Big Blue derived from the TPC-C online transaction processing benchmark test. The eight-core i5 570 is rated at 23,500. That gives a scalability ratio of 1.958. So the scalability that IBM is promising on OLTP workloads that resemble the TPC-C test (as RPG-based ERP systems do) is also coming through on data warehouses--at least those that look like SAP BW.

As for comparisons to other platforms, no one other than IBM has tested a recent server using the SAP BW test. In December 2004, IBM and Novell tested an eight-way xSeries 455 server using 1.5 GHz Itanium 2 processors. The xSeries 455 had about the same load times running DB2 UDB 8.1 on top of SUSE Linux Enterprise Server 8--loading rates of 33.5 and 10.95 million rows per hour on steps one and two of the BW test--the IBM Itanium box running at 96 percent CPU could crank through 98,424 navigation steps. That eight-core server delivered a little better than the same performance as a four-core 1.65 GHz Power5-based i5 550. Even assuming that SUSE Linux Enterprise Server 9 and DB2 8.2 help software performance as much as the additional tweaks to i5/OS V5R3 did (call it 15 percent) and assuming the move to the fastest 1.6 GHz/9 MB cache Itanium 2 processors helps a little (about 9 percent), an eight-way Itanium box will only be able to do about 120,000 navigation steps or so. That gives the i5 570 a 50 percent performance advantage on the SAP BW workload.

In May 2004, server maker Unisys tested a 16-processor ES7000 server on the SAP BW test using the Itanium-based ES7000 (rather than the Xeon-based machines, which were only available as 32-bit boxes at the time). This machine had 16 GB of main memory and 16 of Intel's 1.5 GHz Itanium 2 chips. It could crank through 66.9 rows per hour on the first part of the test, 13.9 million rows per hour on the second part of the test, and then did 154,656 query navigation steps at a processor utilization of 83 percent. This server was running Windows Server 2003 Datacenter Edition and SQL Server 2000, which explains why the server utilization was so low. An eight-core i5 570 did 15 percent more query work than a 16-way Itanium box. Again, with the faster Itanium chips, these two machines might draw even, but the i5 does the same work with half the number of cores. Even assuming that the future SQL Server 2005 from Microsoft delivers stellar performance gains when it comes out in November, it is hard to believe that it has enough performance enhancements to allow an Itanium-based box to draw even on a core-for-core basis with the Power5-based machines.

Here are some even older SAP BW benchmark results. A four-way rx5670 Itanium-based server from Hewlett-Packard running the same test on top of Windows 2003 and SQL Server 2000 was not able to load data as fast on the first test (only 24 million rows per hour), did okay on the second load (10.6 million rows per hour), but only processed 66,852 query navigation tests on the third part of the SAP BW benchmark with the CPUs running at 99 percent of capacity. A 32-bit ProLiant DL740 server with eight 2.8 GHz "Gallatin" Xeon MP processors and 8 GB of main memory running Windows Server 2003 Enterprise Edition, SQL Server 2000, and SAP R/3 4.70 could handle 37 million rows per hour on the first load, 6.7 million rows per hour on the second load, and cranked through 81,792 query navigation steps running at 95 percent of CPU capacity. The i5s smoke these boxes, too.

RELATED STORIES

The i5 Tests Well on SAP Data Warehousing Benchmarks

The i5 Tops SAP Warehousing Benchmark