Modest Gains for X64 Windows and SQL Server on SAP Benchmark

by Alex Woodie

Server partners Microsoft and Hewlett-Packard recently published the results of a benchmark test for SAP's R/3 application running on Windows Server 2003 X64 Edition operating system and SQL Server 2005 database. While the results were not spectacular compared to last year's SAP benchmark results for a similarly equipped X86-based system, they do reflect the modest performance improvements that customers can expect to see when they run 32-bit applications on X64 systems, a Microsoft spokesperson says.

When Microsoft formally launched the X64 edition of Windows Server 2003 at the WinHEC Conference in April, it talked enthusiastically about how all applications--but databases, in particular--would benefit from the increase in addressable memory space that 64-bit computing brings (see "64-Bit Windows Goes Mainstream at WinHEC 2005"). At the show, the software giant wowed the crowd with a live demo that pitted a 64-bit database server against a 32-bit database server in a race to see which one was better at performing an extract, transform, and load (ETL) on 5 million records, a business intelligence workload. While the 64-bit system was pre-ordained to win that battle, few would have predicted that it could have done five times more work than the similarly equipped 32-bit system, which is what it posted in that demo.

While some customers doing high-end data manipulation in a pure 64-bit environment may see large throughput gains of that magnitude, the WinHEC demonstration was really not an indication of the type of performance increase the vast majority of Windows shops should anticipate in the real world. The recent SAP benchmark results indicate the type of performance improvement that average Windows users can expect from the combination of Windows Server 2003 X64 and SQL Server 2005. As it turns out, the performance benefit demonstrated in the SAP benchmark is roughly in line with the 10 to 15 percent performance improvement Microsoft forecasted when running 32-bit applications on 64-bit systems.

The SAP Sales and Distribution (SAP SD) benchmark is used to measure how well servers can handle regular, everyday transaction-oriented workloads, as opposed to the SAP Business Warehouse (SAP BW) benchmark, which is a measure of a system's capability to run process-intensive business intelligence workloads. There are two types of SAP SD tests. The two-tier variant puts the SAP database and the application servers that bang out transactions against it on a single machine, while the three-tier variant uses external application servers and only measures the performance of a single, central database server. The Microsoft software benchmark was three-tier, which means it was set up to measure the performance SQL Server 2005, which is not due to ship until this fall.

The database server used in the SAP SD benchmark test was a four-way Hewlett-Packard ProLiant DL585 server equipped with dual-core AMD Opteron processors running at 2.2 GHz, with 128 KB of L1 cache, 2 MB of L2 cache, and 32.8 GB of main memory. The X64 Edition of Windows Server 2003 Enterprise Edition was used to run the single database server, while on the application side. Thirty-three dialog servers running Windows Server 2003 Enterprise Edition in 32-bit mode ran the 32-bit R/3 application.

According to the published benchmark results, this setup was able to support 18,000 users with an average of 1.87 second response time. These are pretty solid figures, and represent the first time that a Windows system has been able to handle more than 18,000 users, according to Microsoft. In terms of total throughput, this system was capable of handling almost 5.5 million dialog steps per hour, 1.8 million fully processed line items per hour, or 90,950 SAPS. (A SAP is a unit of measure on SAP's SAP-SD benchmark, and is equal to one-twentieth of a fully processed line item per hour.)

For comparisons sake (this is a standardized benchmark test after all), let's see how these numbers stack up against the April 1, 2004, SAP SD benchmark, in which Microsoft's SQL Server 2000 database was run on a four-way HP ProLiant DL580 equipped with single-core Intel Xeon processors running at 3.0 GHz with 20 KB of L1, 512 KB of L2, and 4 MB of L3 on-chip memory, and 8.2 GB of main memory. Aside from the larger on-chip memory, faster processor speeds, and the 2 GB-per-processor limit on main memory imposed by 32-bits, this is, very roughly, about half the hardware used in the SQL Server 2005 test, so we might expect the more recent results to be--again very roughly--about twice as high, all things being equal (which, of course, they never are). The software used in these two tests, SAP R/3 version 4.7 running on the Windows Server 2003 Enterprise Edition, is basically the same. The main difference in software is the addition of the X64 version of Windows (which does bring benefits to 32-bit applications, remember), and SQL Server 2005 instead of SQL Server 2000 used in the earlier test.

The four-way ProLiant DL580 tested in 2004 was able to support 8,016 users with an average of 1.99 second response time. It was able to handle a total of 2.4 million dialog steps per hour, 802,330 fully processed lines per hour, for a total rating of 40,120 SAPS.

As we can see from the comparisons, the ProLiant DL585 running SQL Server 2005 was able to do about 1.27 times more work as the DL580 running SQL Server 2000, in terms of SAPS. A similar improvement is found with the number of users supported, with the newer system supporting 1.24 times the number of users as the SQL Server 2000-based system. So, the SQL Server 2005-based system was able to do a little more than twice as much work as the older system.

Processor utilization was better with the SQL Server 2005 test, but not a whole lot. In the 2004 test of SQL Server 2000, the database server component was running at 91 percent of capacity, whereas the database component in the 2005 test was running at 85 percent. While this doesn't seem like a lot, this is potentially significant, only because HP and Microsoft chose to run the slower 2.2 GHz Opteron processors in the newer test, whereas the faster 3.0 GHz Xeons were used in the 2004 test. Lower processor utilization is one of the benefits Microsoft is touting for 64-bit computing, since the X64 editions of Windows can now use 4 GB of memory per processor (compared to 2 GB with 32-bit versions). If processors are a bottleneck in your database application, you might find significant benefits upgrading to the new X64 versions of Windows Server 2003 and SQL Server 2005 (when it becomes available).

This benchmark raises a few questions. First of all, why would Microsoft want to benchmark a 32-bit application on its new 64-bit gear? According to Microsoft, this setup closely resembles real-world conditions, where migrations to full 64-bit systems will be slow, and users will be running their 32-bit applications on newer 64-bit gear for some time. (In the near future, practically all new server systems will ship with X64 processors from Intel and AMD.)

"Windows Server 2003 X64 Editions and 64-bit SQL Server 2005 allow customers to easily run both 32-bit and 64-bit applications and make the gradual shift to 64-bit computing at their own pace while preserving current investments in 32-bit applications," a Microsoft spokeswoman says. "This benchmark demonstrates the excellent price-performance that can be achieved on standards-based, X86 architecture relative to other larger form factor architectures."

While there is no price component on the SAP SD benchmark (and hence no direct price-performance deductions are possible, despite what Microsoft says), the point that Microsoft is trying to make is that X64 gear is inexpensive compared to RISC/Unix systems. "This benchmark demonstrates the potential for SQL Server 2005-Windows Server X64 database server [workloads] on standards-based, x86 architecture for customers who are looking to migrate or consolidate similar workloads running on Unix," the spokeswoman says. "The benchmark also demonstrates the combined power of X64 and dual-core licensing from Microsoft and its partners."

The point the Microsoft spokeswoman made about pricing and dual-cores is a valid one, because Microsoft decided to treat dual-core processors the same as single-core processors as far as its software pricing goes, effectively giving customers a two-for-one deal on database and operating system license fees. However, Microsoft is not the only one to do this.

Because Microsoft seems bent on talking about pricing and dual cores, in an obvious attempt to distinguish itself from database rival Oracle, which only recently relented its hard-nosed stance on the dual-core pricing issue. (See "Oracle's Multicore Pricing: Right Direction, Not Far Enough") Perhaps we can actually get some hard data and start talking intelligently about price-performance when TPC-C benchmark results for SQL Server 2005 and X64 gear come around.