Big Iron Still Costs Big Bucks
by Timothy Prickett Morgan
The more things change, the more things change relatively to each other in the same directions and the more things look the same. For the past decade and a half, anyone comparing big iron systems such as mainframes and then-nascent Unix and proprietary machines would have come to the conclusion that mainframes were very expensive compared to alternative big iron and therefore represented a great marketing opportunity for vendors of other gear.
Both a cynic and an optimist would say that 15 years from now, someone else will be writing a similar opening paragraph to a similar story. And, of course, perhaps a salesman for IBM's big iron mainframes, with a wry smile and probably with quite a decent commission, too.
That said, even IBM mainframes, like big Unix boxes and the new big SMP boxes that can run Windows or Linux, are all the beneficiaries of Moore's Law, and our analysis of selected big iron systems in November 2001, July 2003, and September 2005 shows that vendors have been able to crank up the performance of their processors and, in some case, cram more cores or L2 and L3 cache onto their processors or chip packages that inevitably leads to big performance increases.
Some of the scalability increases that vendors have been able to squeeze out of their boxes are quite remarkable, and they represent some of the biggest historical gains in performance and scalability that the industry has ever delivered. That is quite a feat, and it is a testament to the idea that not every workload can be moved to a cluster of two-socket servers, and, even if they could be, not every customer wants to use such clusters. For a lot of customers--particularly those who have expertise in big iron systems and who have the big budgets and big egos to match them--throwing hardware, even expensive hardware, at a workload problem is just the easiest and most cost-effective thing to do because the price/performance of the server plus its operating system and relational database management system is only one factor in a buying decision.
Having said that, the price that savvy customers pay for a particular system (by which we mean server hardware, operating system, and database--a terminology that might just be coming back into vogue) absolutely depends on direct and indirect performance and price/performance comparisons with direct and indirect alternatives. The people who say benchmarks and bang-for-the-buck comparisons don't matter are either spending your money (like a bank that holds your checking and stock accounts and passes through systems costs to you) rather than their own or are trying to get you to part with it (such as a server salesman who is trying to tell you to upgrade an existing machine rather than looking at cheaper alternatives).
The analysis in this story is focused only on comparing a few representative big iron configurations from IBM, Hewlett-Packard, Sun Microsystems, Fujitsu-Siemens, NEC, and Unisys. This is by no means an exhaustive comparison, and it is not intended to be one. Rather, it is intended to give a feeling of how different platforms have changed over the past couple of years in terms of their performance and price/performance and how different lines compare to their predecessors and their competition both then and now. The discount levels shown in the tables are estimates based on competitive bids, rumors, and published discounts in various benchmark tests and are not necessarily indicative of all deals striking right now on the street. They are certainly not indicative of situations where a vendor has not put a bid for new servers out to competitors as well as their primary supplier. Any company that does not put such bids out to competitors either has bigger fish to fry or has a recession-proof business and an IT budget that is defying gravity.
All of the big iron machines on the market in 2005 support logical or virtual partitioning as well as physical partitioning in some fashion, and many of them also support multiple and incompatible operating systems on their hardware, sitting alongside one another sharing resources. Most vendors put a proprietary operating system--like z/OS, OS/400, Windows, or OpenVMS--alongside an open systems Unix variant or an open source Linux operating system. Most of the servers aimed at enterprise customers have 16, 32, or 64 processor cores, and some--such as those from Fujitsu-Siemens or Sun Microsystems--scale to 128 or 144 cores in a single system image. The value of such a high processor core count is debatable, particularly since most operating systems can only handle 64 or 128 threads. But all of these big machines can be carved into smaller virtual boxes--and reconfigured on the fly, too--and that is a big benefit that has been long overdue for big iron machines. Dynamic partitions allow customers to buy big machines and sometimes use them just as they would a collection of smaller machines; the idea is to get some of that server sprawl back under control of the central machines again and lower the total cost of administration, even if the cost of a virtual slice of such a server is very high compared to an actual entry or midrange server.
While the big iron machines on the market today have much in common, they have a lot of differences as well. Different machines have radically different architectures, and use different and often incompatible processors, memory and I/O systems, and clustering technologies; they have unique subsystems that further differentiate them from each other. Their basic design philosophies are very different.
But, when it comes to running big databases--which is what most of these big iron machines usually do in commercial data centers--the two questions that customers really want answered are the same across all server platforms: How much oomph do these different machines have, and what on earth do they cost?
Performance and Price Metrics
While there is some publicly available performance and pricing information available for a lot of the big iron machines out there today, the data is still a little thin. Server makers sometimes test their biggest machines, and then leave it to customers to reckon what smaller machines may and may not deliver in terms of performance and price/performance. Sometimes they test a slightly smaller version of their bigger boxes on a workload, presumably because that workload doesn't scale well on their machine. Sometimes vendors give out relative performance metrics (as IBM does for its iSeries, zSeries, and pSeries lines) and leave it to customers to try to figure out how to make this data relevant and comparable to published benchmarks on other platforms.
Despite its detractors and critics, the most widespread benchmark for database performance is still the TPC-C online transaction processing test. But this test is very expensive to run, and not all architectures shine as well on TPC-C as they do on other workloads. This is why Sun Microsystems hasn't run a TPC-C test on its machines for five years. IBM has never published a TPC-C benchmark test on its S/390 or zSeries mainframes, and with the launching of the System z9 machines in July, it did not have a change of heart--for reasons that will become obvious. IBM has similarly not published results for its revamped Power4-based iSeries line in early 2003 or with the Power5-based i5 line in the summer of 2004, and IBM says it has no intention of publishing such information. Big Blue does provide excellent relative performance metrics for its zSeries and iSeries lines, and judging from past benchmarks, it is not all that difficult to get reasonable estimates for TPC-C. No other server vendor provides publicly available relative performance metrics to help customers reckon how a machine will scale as they add processors or to see what performance they can expect if they move from one generation of machine to another.
As I have said many times in the past, there ought to be a law against this.
When Hewlett Packard and Sun Microsystems were trying to convince customers to move from mainframes, AS/400s, and VAXes to Unix servers, they provided relative performance metrics for their servers, because that is what these customers expected and because public benchmarks were not available. We have gained transparency through public benchmarks from SPEC, TPC, and a number of vendors, which is great, but we have lost the ability to reckon the performance, in even a rough sense, across a family of products from a single vendor.
Pricing information for particular configurations can be difficult to obtain as well, and getting a sense of the discounting levels in the big iron market is even more problematic. Traditionally, the price/performance of a large system or server has always been better than the middle of any product line and generally equal to, if not better, than the entry machines in that line. This is a normal bell curve for price/performance. Things go a little awry in terms of discounting when the economy is bad and when multiple big iron players are brought in on a deal. Discounting can be very heavy on the initial installs of these big servers, but such discount levels are not indicative of the pricing levels customers can expect as they add components such as extra processors to their machines. A 45 percent or 50 percent discount on a multi-million dollar big iron Unix server might help cushion the blow when moving from one Unix server to another server that has a different Unix operating system and different underlying administration tools. But customers have to be careful and look at upgrade prices and performance scalability within any product line they adopt. You might save some money on the front end of a deal, and end up paying dearly when it comes time to upgrade. You can bet one thing: Unless you have upgrade pricing woven into your initial contract with a new vendor, processor upgrades are not going to come in at that same 45 percent or 50 percent discount level.
Another factor that makes discounting tough to reckon is the relative thinness of deals in this market. While billions of dollars of big iron are sold each year, there are only several thousand deals. Wholesale replacement of one platform with another is dramatic and rabidly reported on in the press when it happens, but it is a relatively rare occurrence even if the possibility of losing an account to a competitor does make every big iron player paranoid. Finally, discount levels on hardware are in large measure dictated by how much money a customer wants to spend on software and services as part of the deal. The more software and services a company wants or needs (the two are not the same thing), the steeper the discounting.
Unfortunately, not every vendor runs standard benchmarks on their machines, so in a lot of cases the OLTP performance (in transactions per minute) have been estimated in the absence of real TPC-C OLTP benchmark test results. The TPC consortium frowns upon this sort of thing, and vendors hate it, but because of the thinness of the benchmark data and the breadth of most vendors' product lines, customers have little choice but to make such estimates. That is why we feel justified in making these estimates, even though philosophically we are opposed to such estimates for exactly the reasons the TPC has expressed. If vendors don't run tests, they will let people make estimates, and they get the benefit of having a test result without actually going through the rigorous and expensive testing and auditing procedures that are part and parcel of a TPC-C benchmark test.
The OLTP performance shown in the two tables is the maximum expected performance of the central electronics complex in each server, provided that memory and I/O are not the bottlenecks in the system. The pricing shown is for a reasonable base hardware configuration and the operating system and database licenses necessary to support the number of TPC-C users on mainframe, OS/400, Unix, or Windows platforms. Wherever possible, we have tried to price Itanium machines with Unix or Linux operating systems. Windows may rule the roost in the entry-server space, but it is not even close to dominant in the datacenter. That said, the performance and price/performance of Windows, Unix, and Linux on machines that support two or three of these platforms has been very similar. There is generally only a few percent difference in performance.
The pricing comparisons for Unix, Linux, and iSeries servers include operating systems and databases and are priced using processor-based, unlimited user licenses. The Linux, Windows, and Unix machines from 2001 are configured with Oracle8i Enterprise Edition; in 2003 they have Oracle 9i Enterprise Edition, and in 2005 they have Oracle 10g Enterprise Edition. While many midrange and some X86, X64, and Itanium servers in the big iron class are tested running Windows and SQL Server, there is no reason to suspect that Oracle performance would be appreciably different on these machines. Vendors use SQL Server on benchmark tests because it can be half the cost of Oracle for some configurations. But SQL Server does not run on Unix platforms, and is a big unknown at this upper echelon of the server market. That is why we have opted to put the Oracle database on all of the machines possible in our big iron comparison.
It is not possible to run Oracle databases on IBM's iSeries machines, which can only run its own DB2/400 database, which is actually woven into that machine's OS/400 or i5/OS operating system. (They are the same thing, just the name was changed in 2004 with the launch of the Power5-based i5 servers.) IBM has two versions of i5/OS and its DB2/400 database: Enterprise Edition allows RPG and COBOL programs to make use of the 5250 green-screen protocol for OLTP workloads, for which IBM charges a premium, while Standard Edition does not support 5250 and costs less.
The comparisons for the IBM zSeries 900, zSeries 990, and System z9 mainframes do not include the cost of the z/OS operating system or IBM's DB2 variant for mainframes. The z/OS operating system and DB2 database used to be sold with a basic one-time charge that was roughly equal to renting it for several years, but now IBM is only selling these programs (with a few exceptions on small mainframes) under usage-based, monthly rental pricing schemes. You need a PhD in numerology to figure out what z/OS and DB2 cost on the zSeries, but the point is moot. The cost is hundreds of thousands of dollars per month (or more) on these big mainframes, and even without these base software charges, mainframe server pricing--even at a respectable discount--is very high compared to big Unix servers.
These comparisons between these big iron machines assume that the boxes are going full-out doing database processing. That is, after all, what the three-tier TPC-C test is doing. But mainframe, OS/400, and Unix customers run many different workloads on their machines concurrently, not just a database. They have sophisticated workload managers that not only make this possible but also push processor utilization into the 70 percent or higher range.
Where They Stood, and Where They Stand Today
If you want to know why many of the major server vendors are trying to expand into software or services, it doesn't take but a cursory glance at the data for 2001, 2003, and 2005 to see why. Being in the server business means having to sell more and more capacity each year to stay in the same place. This is one of the roughest businesses on the planet, and that is why the industry has contracted from hundreds of server and system platforms to a relative handful. In general, the price of server capacity has declined by a factor of two to three between July 2003 and September 2005.
Contrast this with software. The other thing that stands out is the fact that the price for a high-end Oracle database has not changed one penny--at least officially--for four years. The only change, and one that just happened a few months ago, is that Oracle is pricing its databases on dual-core processors with a 25 percent discount on the second core in a chip complex that has two cores. This helps IBM's Power4 and Power5 platforms, Sun's UltraSparc-IV platforms, and HP's PA-RISC platforms today, and will help cushion the blow for the future dual-core "Olympic" Sparc64 VI chip from Fujitsu (to be sold in servers by both Fujitsu-Siemens and Sun) and dual-core "Montecito" Itaniums from Intel, which will be used in HP's Integrity line of servers as well as in big boxes from NEC, Unisys, Fujitsu-Siemens, and a few others.
To start with, click here to see the baseline configurations from a report I put together in November 2001.
My initial reason for doing this comparison in late 2001 was to compare various servers in roughly the same performance class. (I was paid by a big bank as a consulting gig to do this; I didn't just do it for fun, but it was fun.) At the time, IBM, HP, and Fujitsu-Siemens offered slightly more scalable machines, but the biggest Unix machines offered roughly the same performance, with IBM taking the lead with its Power4-based systems. This is a lead in performance that IBM has held from that moment on, and it is very likely to be a lead IBM holds for the foreseeable future unless something radical changes in the chip and server business--like Sun's future "Rock" processors come to market with lots of power or Intel gets quad-core Itaniums with lots of oomph to market. Or, if someone decides to build big boxes on Opteron (if it is even technically possible to scale that far with Opteron chips.)
While the performance of these big boxes were more or less the same, with IBM's zSeries mainframes and Sun's Sun Fire servers lagging in terms of performance and IBM's pSeries and iSeries leading, the cost of acquiring one of these boxes varied dramatically across vendors. Sun's Sun Fire 10000 server was the first big Unix box, and Sun was still egoistically charging mainframe-class prices for it in late 2001. And even the then-new Sun Fire 15K servers, using 900 MHz UltraSparc-III chips, were still a lot more pricey than IBM's and HP's Unix kit. IBM's iSeries machines running OS/400 Enterprise Edition were also very high priced compared to Unix boxes, even after big discounts. As the economies of the world went into recession and the IT market dealt with the massive amount of capacity it shipped between 1998 and 2001 for the ERP, Y2K, and dot-com bubbles, server makers cranked up the clocks, discounted like crazy, and somewhat paradoxically delivered even more capacity to server buyers needing big iron. The idea was that bigger was better--better for customers, who should consolidate, and better for vendors, who wanted to get as many big, profitable deals as they could and lock out their competition from accounts.
I took another snapshot of the high-end of the server market in July 2003, and the performance and price/performance increases since late 2001 were, in many cases, just stunning.
As is obvious from the July 2003 table, IBM put a lot of water between itself and other big iron vendors with the 32-core pSeries 690 "Regatta" server, and even beat out the "T-Rex" zSeries 990 mainframe by a considerable margin in terms of performance--740,000 TPM on the TPC-C test versus an estimated 450,000 on the zSeries 990, with both machines having 32 cores dedicated to OLTP work. In 2004, prior to the launch of the Power5-based servers that summer, IBM and HP were both pushing just about 1 million TPM on the TPC-C test with their respective 32-core machines. (HP did a lot of tuning to boost performance on the Superdomes at this time, as did IBM.) Sun could deliver about 475,000 TPM, but it needed an incredible 72 processor cores to hit that level, and even with dual-core UltraSparc-IV chips (which were supposed to ship in 2004 or so), Sun would have been beaten by IBM and HP in the TPC-C game. That is why Sun dropped out of the game. The most scalable Sparc platforms at the time came from Fujitsu-Siemens, which could crest above 522,000 TPM with a 48-processor PrimePower 2500 using its then-new 1.3 GHz Sparc64 V processors. The "Madison" Itanium 2 processors from Intel were yielding very good performance, with 32-core systems in roughly in this range, but the Power4 chips and the pSeries design eventually delivered about twice the performance per core.
IBM delivered triple-digit performance gains (as reckoned by comparisons of big boxes) in its pSeries and zSeries lines between 2003 and 2005, and very high double-digit gains with the iSeries. Within those customer bases, this must have seemed like a great deal at first. But the actual improvements in price/performance for the pSeries line were much more modest, although for the big iSeries machine using 5250 software the price changes were dramatic because IBM stopped gouging these customers so deeply. HP actually showed much better price/performance improvement during this time with its Itanium-based Superdomes compared to its PA-RISC Superdomes, which demonstrates why HP made that move to begin with. Even Sun delivered better relative improvements in bang for the buck during this time, as did Unisys with its Itanium-based ES7000s compared to its Xeon MP-based ES7000s from two years earlier. To put it bluntly, IBM raised the class average on performance, while its competitors tried to make up for past lost ground on price/performance that IBM had stolen with its 2001 generation of Power4 servers.
Fast forward to September 2005, and take a look at the server landscape. As you can see, IBM is at it again, just hammering the competition with performance and price/performance improvements with its pSeries and iSeries "Squadron" Power5-based servers. But, this time, HP's Superdomes are also delivering triple-digit gains compared to 2003, and if Sun holds its prices steady when it delivers the "Panther" UltraSparc-IV+ chips in the Sun Fire 20000 and 25000 servers later this year--and there is no reason to believe that it will not, since it just did that a few weeks ago with its midrange Sun Fire line--then Sun will be right in the same range, delivering triple digit gains and, just perhaps.
I estimate that Sun will also hit 2 million TPM with its high-end Sun Fire 25K servers for the first time, but it will take a fully loaded server with 144 1.5 GHz UltraSparc-IV+ processor cores to accomplish the task. And even with that, largely because of the effects of having more than twice as many processor cores in the server compared to IBM's 64-core i5 595 server, Oracle costs will still make the Sun box twice as expensive as the IBM box for customers who opt for processor-based pricing. Of course, customers that have enterprise site licenses for Oracle databases and that do not pay based on the capacity of their servers will find that for the first time in five years, the base IBM and Sun Unix servers at the high end will deliver about the same price/performance--again, just for the raw iron. This assumes that IBM, because it is delivering such high price/performance, is only discounting hardware at around 25 percent on the street, while Sun keeps Sun Fire 20K and 25K server prices steady as it doubles performance and continues discounting at 30 percent on the hardware. HP has to discount its Integrity Superdome servers at 40 percent for a street price that gets its servers in the same range as the p5 595s, but then the Oracle prices drive up the overall base system cost. Add to that the fact that IBM is pushing a lot more performance out of its Power5 machines and you can see that HP must be hopping mad--as in nearly mad enough to think of hopping to Opteron for the Superdomes, but it cannot do that because of operating system, middleware, and application software issues--that the dual-core Montecito chips, which were supposed to be here in mid-2005, are not here. HP needs Montecito to compete against Power5. It is that simple.
And so do Fujitsu-Siemens, NEC, and Unisys, who keep relatively low profiles and who go after the business they can get with their somewhat less scalable big iron machines. They needed to double the performance of their Itanium-based machines as well in 2005, and now they have to wait until 2006.
The big mystery is what Fujitsu-Siemens and Sun will do in terms of pricing and performance with the Olympic processors and their associated "Jupiter" servers. Fujitsu-Siemens has been telling customers the dual-core Sparc64 VI processors will deliver a little more than twice the oomph, chip for chip, as the current 1.3 GHz and 1.35 GHz single-core Sparc64 V chips. There is speculation that the Jupiter machines will have 64 sockets, compared to the 128 sockets in the current PrimePower 2500s. That would seem to indicate that while the per-chip performance for the Jupiter servers will double, the ultimate scalability of the machines may not because there will be the same number of processor cores in the box. If this turns out to be true, then a 64-socket Jupiter machine with a 2.4 GHz dual-core Olympic chip should hit about 1.3 million TPM on the TPC-C test. That's where IBM and HP were in terms of raw performance at the end of 2003, and that is about where Sun is today with its Sun Fire 25K server using the "Jaguar" UltraSparc-IV dual-core processors running at 1.2 GHz.
Of course, TPC-C performance--both real and estimated--is not everything. All of these servers excel at running many applications, and, most importantly, they continue to offer performance increases to the customers who use these platforms. That is, after all, the real name of the game in the server racket, nine times out of ten. If you can keep a customer happy, they don't want to switch server platforms.
I didn't include every machine I compared in the summary tables, but if you want to see all the raw data, all you got to do is click here.
November 2001 Big Iron configuration data
July 2003 Big Iron configuration data
September 2005 Big Iron configuration data
Raw data for all three tables, which has a lot more configurations in it