TPC Benchmarks Track Energy Usage Now, Too

February 8, 2010 Timothy Prickett Morgan

The members of the Transcation Processing Council last week ratified and announced a new add-on feature of the popular TPC suite of tests for measuring online transaction processing and data warehousing performance. For years, the TPC tests have measured performance and bang for the buck. But now, thanks to the TPC-Energy specification, vendors now have a consistent way to measure, audit, and report how the systems they test consume energy as they run benchmarks.

According to Mike Nikolaiev, who is the manager of the systems performance unit at Hewlett-Packard and chaired the TPC-Energy committee, this is the first specification that was ratified unanimously in the history of the TPC. So there appears to be universal enthusiasm for measuring energy consumption by systems among the 24 server, storage, operating system, and database vendors who make up the consortium. My guess is they are all hoping they can prove they give the best performance per watt or the best dollar per transaction or query per watt, and this is another arrow in the sales quiver.

And, more significantly, just because vendors ratify a spec doesn’t mean they will use it. The TPC-Energy spec is an overlay of the existing TPC-C online transaction processing test (which has been used since 1992) and the relatively new (since early 2007) and still fairly unpopular TPC-E OLTP test; energy consumption can be added to TPC-H data warehousing benchmark tests, too.

As previously reported in The Four Hundred, the Standard Performance Evaluation Corporation started adding power consumption measurement to its Web serving and Java application benchmarks. While this is all well and good, the SPEC tests are really CPU-level tests, not system level tests, and they also do not require full parts lists and pricing for the entire system under test. Moreover, the TPC has very stringent and independent auditing requirements. Both tests have their limits. The SPEC tests are relatively easy and inexpensive to run, while the TPC tests are expensive and, with the TPC-C test in particular, some vendors have claimed that the application is too simple to stress today’s very powerful systems and others point fingers and say that their peers are gaming the benchmarks. (What? An IT vendor that exaggerates performance and shows unrealistic configurations or discounts? I am shocked, just shocked. . . .)

To help cut down on gaming on power measurement on the TPC tests, the consortium came up with a single set of tools for monitoring energy usage and capturing the data as systems run their tests called, appropriately enough, the Energy Measuring System. As part of the energy spec, vendors have to have the air intake to their server and storage systems at a minimum ambient temperature of 20 degrees Celsius, which means you can’t supercool the data center itself to help the server work harder and run cooler as it does so.

The TPC expects the first tests using the TPC-Energy spec to come out in the next month or two. It would be great to see IBM‘s i/OS machinery be the first to use the TPC-Energy spec, which would be particularly appropriate since the AS/400-E systems from 1992 were the first servers ever to use the TPC-C OLTP test. I would love to see a test that showed how much less energy green-screen, compiled RPG applications use compared to uncompiled Java and PHP workloads on the test. And how much less iron it takes to do the work. I think this would be truly hilarious–and perhaps even useful.

This Issue Sponsored By

Table of Contents

Content archive

Recent Posts

Subscribe

Pages

Search