The Top 500 Super Ranking Now Counts Watts as Well as Flops

Published: June 24, 2008

by Timothy Prickett Morgan

The 31st edition of the semi-annual Top 500 listing of supercomputers in the world was announced last week at the International Supercomputing Conference 2008 event in Dresden, Germany, and the list will be noteworthy for two reasons. First, this will be the first time that a supercomputer of any make, vendor, or architecture has broken through the petaflops performance barrier, a feat accomplished by the "Roadrunner" hybrid Opteron-Cell machine created for the U.S. government's Los Alamos National Laboratory by IBM. And second, and perhaps foremost, this will be the first ranking where watts are being counted for big machines in a manner that is as consistent as the Linpack Fortran benchmark that is the touchstone for performance.

Such a flops per watt ranking is a long time coming, particularly with supercomputing centers requiring multiple megawatts of juice to power their servers, storage, networks, and air conditioning facilities. Performance always comes at a price that is larger than the hardware itself, and it is high time that comparisons of architectures show at least some of the penalties in heating and cooling that accompanying them.

The BlueGene design created by IBM nearly a decade ago in the labs in upstate New York was the first time a major computer maker publicly talked about trying to hit petaflops computing and the ridiculous power and cooling issues involved. The resulting BlueGene/L commercial computers, according to the supercomputer experts who put together the Top 500 list--Erich Strohmaier and Horst Simon, computer scientists at Lawrence Berkeley National Laboratory, Jack Dongarra of the University of Tennessee, and Hans Meuer of the University of Manheim--said that BlueGene/L, which is located at Sandia National Lab and which set the ceiling for performance per watt in supercomputing until recently, is rated at 210 megaflops per watt, but has already been surpassed by the Roadrunner super not only in performance, but in performance per watt, which comes in at a staggering 488 megaflops per watt. The updated BlueGene box, called BlueGene/P, is rated at 376 megaflops per watt, and machines based on Intel's quad-core "Harpertown" Xeon 5400 processors are not far behind. They cite a cluster based on IBM's BladeCenter HS21 blades using low-power Harpertowns coming in at 265 megaflops per watt, a cluster of Silicon Graphics Altix ICE 8200EX servers providing 240 megaflops per watt, and a Hewlett-Packard Cluster Platform 3000 using BL2x220 double-density blades hitting 227 megaflops per watt.

The machines at the top of the list, which you can view here, are generally speaking the most efficient ones on the list, and this stands to reason since they are the newest boxes deploying the latest technologies. Among the top 10 machines in the list, the average machine consumes 1.32 megawatts and has a power efficiency of 248 megaflops per watt. (Roadrunner is bringing up the class average, bigtime.) Across the 50 biggest machines in the list, average power consumption falls to 908 kilowatts and average power efficiency falls to 193 megaflops per watt, while across the whole list of 500 machines, average power consumption falls to 257 kilowatts and efficiency falls further to 122 megaflops per watt.

For now, of course, what people are still bragging about is performance, and by breaking through the petaflops barrier, IBM once again is able to get bragging rights for this $133 million machine at Los Alamos. (See Beep, Beep: Roadrunner Linux Super Breaks the Petaflops Barrier for details on this innovative machine, which is a blade design that mixes Opteron LS21 and Cell QS22 blade servers). The BlueGene/L system, which has held the top spot since November 2004 and which has been upgraded several times, falls to number two on the list, at 478.2 sustained teraflops with a staggering 212,992 PowerPC cores. Three BlueGene/P kickers to this box--number three located at Argonne National Lab and rated at 450.3 teraflops using 163,840 processor cores, number six located at Forschungszentrum Juelich in Germany and rated at 222.8 teraflops using 65,536 cores, and number nine located at IRDIS in France and rated at 112.5 teraflops using 40,960 cores--helped IBM take dominant share of the performance in the top ten systems. IBM is the leader with 210 machines on the 31st edition of the Top 500 list, but the number of machines Big Blue has on the list has actually dropped from 242 machines six months ago.

Hewlett-Packard, by virtue of the substantial supercomputing business it inherited seven years ago through its acquisition of Compaq, closed the system count gap with IBM a little this time around, with 183 systems on the list, up from 166 machines on the November 2007 list. The largest HP machine on the list is a cluster based on BL460c blades that is installed at Tata Industries in India, which comes in at number eight in the rankings; this box uses Intel's quad-core "Clovertown" Xeon 5300s, and sports 14,352 cores running at 3 GHz, delivering 132.8 teraflops of sustained performance.

For the first time in a long time, Sun Microsystems is making a strong showing on the Top 500 list, with the "Ranger" Constellation System installed at the University of Texas, which uses Sun's Opteron blades and a massive InfiniBand switch that it designed to deliver 326 teraflops of performance with 62,976 cores. Ranger is ranked number four on the list, and is followed by the Cray XT4 Opteron machine installed at Oak Ridge National Laboratory, nicknamed "Jaguar," which has 30,976 cores and is rated at 205 teraflops of sustained performance. SGI had two machines in the top 10, both of them Altix ICE boxes using Xeons, not the Itanium-based Altix 4700s. Number seven is an SGI box named Encanto installed at the New Mexico Computing Applications Center that has 14,336 dual-core Xeons and is rated at 133.2 teaflops and number 10 is a similar box installed at oil giant Total-Fina in France that uses quad-core Xeons (a total of 10,240 cores) and is rated at 106.1 teraflops sustained. Cray has 16 machines on the list, SGI has 22, and Sun has 5 if you count a big one in Japan it shares with NEC. Dell has 27 machines on the list, and the remaining smattering of machines are from other players, sometimes niche super makers, or homegrown, if you can believe it.

Significantly, the turnover rate on the Top 500 list is the largest seen in the 16-year history of the list. The 500th system on the list was ranked at 200 on the list only six months ago, so several hundred government, academic, and corporate supercomputer center sites have bought lots of new capacity.

In terms of platform, Linux continues to dominate the Top 500 list, with all of the top 10 systems running Linux and a total of 427 machines running the open source Unix-alike platform. Unix is used on 28 machines in the ranking (if you count Mac OS as Unix, and I do) and Microsoft's Windows Compute Cluster Server even has 5 machines on the list now. Some 40 machines on the list run mixed operating systems, and these are generally a mix of Linux and Unix. That gives Linux an 85.4 percent share of machines, 76.4 percent of total peak teraflops of capacity, and 58.3 percent of processor cores. The mixed systems may only represent 8 percent of systems on the list, but they account for 16.9 percent of peak performance and 37.1 percent of cores. So don't go thinking that the Mac OS and Windows boxes are somehow making up more than their share of aggregate performance or cores. They are not.

By processor type, quad-core chips are making significant gains, with 283 machines already using them and another 203 using dual-core processors. Only 11 machines on the list still use single-core chips, and only 3 machines use 32-bit X86 chips. On the June 2008 list, 16 have Itanium chips, while 356 use 64-bit Xeons and 55 use 64-bit Opterons from Advanced Micro Devices. Only one machine uses NEC's vector processors, only one uses Cray's vector processors, and 68 use one or another variant of IBM's Power chips. Intel has a 75 percent share of Top 500 systems this time around, up from 70.8 percent six months ago.

Gigabit Ethernet is by far the most popular interconnect used in the Top 500, with 285 machines on the list, followed by InfiniBand, with 120 systems and a lot of them near the top of the list where expense is less of a concern. The remaining machines are a mix of NUMAlink (SGI), SP Switch (IBM), proprietary crossbar, Myrinet, Quadrics, or mixed interconnects.

RELATED STORIES

Beep, Beep: Roadrunner Linux Super Breaks the Petaflops Barrier

Blade Servers Make It to the Top HPC Sites

Top 500 Supers: Moore's Law Is Alive and Well

Microsoft Back on the Top 500 List of Biggest HPC Systems

Dual-Core Processors Begin Takeover of Top 500 Super Ranking

Top 500 Supers: Brace Yourself for Petaflops Systems

Linux Clusters Continue to Expand in Top 500 Supers Ranking

Top 500 Supers List Dominated By Exotic Clusters

Top 500 Supers List Dominated by Teraflops-Class Machines

Linux, X86 Clusters Take Over Top 500 Supercomputer Ranking

Top 500 Supercomputer Ranking Gets Top Heavy

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot