Top 500 Supers: Moore's Law Is Alive and Well
Published: July 10, 2007
by Timothy Prickett Morgan
Call it a rite of summer. The International Supercomputing Conference was going on in Dresden, Germany, as we were publishing the prior edition of this newsletter, and each year this is when the first of the Top 500 supercomputer lists ranking the most powerful number-crunchers in the world comes out. (In the fall, it is the Supercomputing event in Silicon Gulch that has the second ranking.) The current Top 500 ranking shows a number of things: multicore X64 processors are popular, Linux is king in HPC, and Moore's Law is alive and well when it comes to supercomputers, thanks in large part to high-speed and affordable interconnection electronics.
The Top 500 supercomputer ranking is put together by Hans Meuer of the University of Mannheim, Erich Strohmaier and Horst Simon of Lawrence Berkeley National Laboratory, and Jack Dongarra of the University of Tennessee. Dongarra invented a Fortran matrix math benchmark called Linpack many years ago, and the Top 500 ranks the unclassified supercomputers of the world using the Linpack test. Being at the top of the list, or in any way dominating it (for instance, in terms of shipments or aggregate customers) has been and continues to be a big deal to the makers of vector supercomputers, parallel clusters, and other exotic HPC gear. In fact, it would be fair to say that the power of supercomputers at the system level is growing a bit faster than the performance of individual processors, which has slowed a bit since the last decade because it is harder to increase performance with multiple cores while staying in the same power envelope than it is to just crank the clock on a single core chip and create more heat.
The aggregate computing power represented by the current Top 500 list is now 4.9 petaflops, almost double from the 2.8 petaflops level in the June 2006 report. Just to get on the Top 500 list this time around, a supercomputer had to have more than 4 teraflops of computing power; only six months ago, the entry point was just over 2.7 teraflops and a year ago a system only needed 2 teraflops. The current list has, in fact, had the highest churn rate among machines in the 15-year history of the Top 500 ranking. The machine that is ranked 500th on the recent list was ranked at 216th only six months ago. Governments, academic research centers, and corporations are investing heavily in big systems, and they are doing so because InfiniBand and Gigabit Ethernet interconnects are relatively inexpensive, Linux is dominant in HPC and cross-platform, and X64 processors deliver great bang for the buck. To be sure, there are still plenty of exotic machines on the Top 500 list--it would not be supercomputing if there were not.
IBM's Blue Gene/L Power-Linux supercomputer, built for the U.S. Department of Energy for Lawrence Livermore National Lab, retains the top spot on the list for the fourth time, with a Linpack rating of 280.6 teraflops. Blue Gene/L has a stunning 131,072 dual-core PowerPC processors. IBM utterly dominates the Top 10 systems, with six out of 10 machines, including a 91.3 teraflops Blue Gene box IBM owns as well as two new ones, a 82.2 teraflops Blue Gene installed at the State University of New York's Stony Brook campus and a 73 teraflops machine installed at Rensselaer Polytechnic Institute (also in New York). The most powerful supercomputer in Europe remains the MareNostrum cluster, which is installed at the Barcelona Supercomputing Center in Spain and which ranks number nine on the list with a rating of 62.6 teraflops. The Power5+ ASCI Purple parallel RISC/Unix box at Lawrence Livermore is number six on the list, with its 12,208 processors yielding 75.7 teraflops of power. IBM has a total of 192 machines on the Top 500 list, or a 38.4 percent share, and has about 42 percent of the aggregate computing represented on the list.
After some years of hard times, Cray's investment in its "Red Storm" Opteron-Linux architecture is paying off. The Red Storm machine at Sandia National Laboratories is now rated at 101.4 teraflops, and a companion XT4/Xt3 cluster based on the same architecture installed at Oak Ridge National Laboratory is rated at 101.7 teraflops. These machines are ranked at number two and three on the list. Cray has a total of 11 machines on the list, or about 2.2 percent of machines, but these two big boxes and a number of others bring Cray's share of the Top 500 petaflops pie up to 7 percent.
Dell and Silicon Graphics made it into the Top 10, Dell at number eight with a 62.7 teraflops X64-InfiniBand cluster running Linux at the National Supercomputing Center and SGI at number 10 with an Itanium-InfiniBand cluster rated at 62.6 teraflops running at Leibniz Rechenzentrum in Germany. Dell has a total of 23 systems on the list, or 4.6 percent of the base of 500 machines, and about 9 percent of the aggregate 4 petaflops represented by the current list.
Hewlett-Packard has not been anywhere near the top of the list in a long time, and the ASCI-Q supercomputer installed by Compaq at Los Alamos National Laboratory was the last box that could be said to have an HP label that ranked near the top of the list, and that machine, now ranked number 62 on the list with 13.9 teraflops, is the biggest HP box on the list. But don't get confused. HP is a big player in the HPC market, and has the most machines of any vendor on the Top 500 list, with 202 boxes of a 40.4 percent share. HP doesn't have big boxes at the top raising its teraflops average, so HP's share of the aggregate capacity on the Top 500 list is only about 24 percent.
There are a number of other players who have machines on the list, including Sun Microsystems, Fujitsu, NEC, Linux Networx, Appro International, and there are even five self-made supers on the list.
In terms of processor architecture, X64 processors dominate the current Top 500 list as they did a year ago. There are 231 machines using Intel's X64 chips and another 107 using Advanced Micro Devices' X64 processors; only 28 machines are based on Intel's Itanium chips, and there are still 28 machines using 32-bit X86 processors. Of the X64 machines, 91 of the 107 machines using Opterons have dual-core Opterons and 204 of the 231 machines using Xeons have dual-core chips; 19 boxes using Intel chips are using the company's quad-core processors already. Another three boxes are using dual-core Itanium processors, by the way. Multicore is the norm in HPC now, but then again, you'd expect that given the multi-node nature of clusters. There are 85 machines based on the Power architecture, and three supers on the list based on Fujitsu Sparc64 chips, plus a handful of machines using Alpha or PA-RISC chips and a few Cray vector boxes.
While InfiniBand has received a certain amount of grief over the years, supercomputing centers are adopting it now, with 128 machines on the list using InfiniBand. But the much less costly Gigabit Ethernet interconnect is still used on 206 machines on the list. Myrinet had 46 boxes using its own interconnect on the list, and Quadrics had 11, IBM had 36 using its SP switch, and the remainder are a mix of proprietary interconnects and crossbar architectures.
As has been the case for many years, Linux remains the dominant operating system in the Top 500 list, with 389 machines, or 78 percent, running this open source operating system. Many supercomputer centers have their own variants of Linux, and some run modified yet supported versions from their hardware providers. There are 64 machines running one flavor or another of Unix, and 67 if you count Mac OS X as Unix (and you should). There are two Windows-based clusters, and 42 machines that run a mix of operating systems--and one of those operating systems in the mix is always a Linux and the other is a variant of Unix. If you want to be fair, Linux is represented on 86 percent of the machines and Unix is on 22 percent, with Microsoft getting 1 percent. Microsoft will probably not get a big slice of the Top 50 systems any time soon, but the bottom part of the list could start turning to Windows Compute Cluster Server, and more importantly, the a large number of future machines with teraflops of capacity that would never make it onto the Top 500--or maybe not even the Top 2000--list could nonetheless give Microsoft a real foothold in HPC. Researchers who know their field and do not want to learn Linux probably know a little Windows, and people tend to stick with what they know. That's what Microsoft was counting on when it got into the server racket more than a decade ago, and this strategy has worked marvelously.
The United States had 281 of the Top 500 systems within its borders--and very likely a number of other classified and very large machines that will never make it on to the list but nonetheless still exist. Europe had 127 systems, continuing to rebound after a slump, with the United Kingdom's 43 machines pulling well ahead of Germany's 19 on the list. Japan had 23 machines on the list, falling by seven boxes, and China fell to 13 machines, down by five.
Which brings me to a final set of questions. So when does China build the largest supercomputer in the world, just to freak everybody out? And will any companies based in the United States, Europe, or Japan be allowed to bid on such a deal, and if they are allowed to bid, will they win the deal? And if they win the deal, will their own national governments make a big stink about it?
Dual-Core Processors Begin Takeover of Top 500 Super Ranking
Top 500 Supers: Brace Yourself for Petaflops Systems
Linux Clusters Continue to Expand in Top 500 Supers Ranking
Top 500 Supers List Dominated By Exotic Clusters
Top 500 Supers List Dominated by Teraflops-Class Machines
Linux, X86 Clusters Take Over Top 500 Supercomputer Ranking
Top 500 Supercomputer Ranking Gets Top Heavy
Post this story to del.icio.us
Post this story to Digg
Post this story to Slashdot