Top 500 Supers: Brace Yourself for Petaflops Systems

Published: July 11, 2006

by Timothy Prickett Morgan

The International Supercomputing Conference was in full swing in Dresden, Germany, just before the holiday, and the Top 500 listing of supercomputers, which is the equivalent of the Billboard chart ranking for popular music, is always the star of any show it is timed to coincide with. Two weeks ago at ISC2006, the situation is no different. Server vendors and their governments are still vying to be on the top of the flops charts, and the academic, government, and private supercomputer centers are still consuming vast amounts of number-crunching power.

This is the 27th semi-annual supercomputer ranking put together by the University of Manheim and the University of Tennessee. The ranking is based on the peak and sustained performance of the 500 most powerful supercomputer systems in the world using the Linpack Fortran benchmark test developed by Jack Dongarra of the University of Tennessee many years ago. The Top 500 list is compiled by Hans Meuer of the University of Mannheim; Erich Strohmaier and Horst Simon of Lawrence Berkeley National Laboratory; and Dongarra of the University of Tennessee. They have the security clearances necessary to measure and rank the biggest systems in the world, many of which are classified.

Following along with Moore's Law, the total combined power in the Top 500 ranking has increased substantially in the past 12 months, rising by 65 percent to 2.79 petaflops. (That's millions of gigaflops.) To even get on the list this year, a supercomputer center has to build a machine with over 2 teraflops of raw computing capacity, double that from a year ago. To get into the Top 100 takes more than 4.7 teraflops, and to get into the Top 10 takes 35 teraflops. There is tremendous churn in the Top 500 list, with 174 systems in the June 2006 ranking being installed this year and another 231 machine being installed in 2005. The advent of ever cheaper and more powerful servers, faster interconnects, and sophisticated cluster and workload management software is empowering supercomputer centers to put a lot of computing at the disposal of their researchers.

The biggest, baddest box on the list is the Blue Gene/L Linux-Power supercomputer by IBM for the U.S. Department of Energy at Lawrence Livermore National Laboratory. This machine has 131,072 PowerPC processors and a peak rating of 367 teraflops and a sustained rating of 280.6 teraflops on the Linpack test. This monster of a machine is expected to stay at the top of the supercomputing charts for the next couple of lists. But both Cray and IBM are racing to hit 1 petaflops of peak performance, and it will be interesting to see who hits it first--and if some upstart tries to steal the crown away, as Intel did a decade ago with the ASCI Red massively parallel X86 machine and NEC did a few years ago with the parallel vector machine called Earth Simulator.

IBM had the number two slot on the list with a homegrown Blue Gene machine, called Blue Gene Watson, which has 40,960 processors and a sustained rating of 91.3 teraflops, and the number three slot with the ASCI Purple machine that sits beside Blue Gene/L at LLNL, which has 12,208 1.9 GHz Power5 processors and a peak rating on the Linpack test of 75.8 teraflops. The number four machine is the "Columbia" Itanium cluster built for NASA's Ames Research Center using its Itanium-based Altix machines from Silicon Graphics and InfiniBand interconnect, which has a sustained rating of 51.8 teraflops and has 10,160 1.5 GHz Itanium processors (this machine is noteworthy in how efficient it is, by the way). The number five machine in the list is also an Itanium-based machine, the Tera-10 supercomputer built by Bull for the Commissariat a l'Energie Atomique in France. This super is based on Bull's NovaScale Itanium servers, and has 8,704 processors lashed together using Quadrics interconnect. Tera-10 is rated at 42.9 sustained teraflops.

Some of the next five supercomputers used to place higher on the list, but have moved down. Some are totally new systems. Sandia National Laboratories' "Thunderbird" super, a cluster made from Dell PowerEdge 1850 servers 9,024 3.6 GHz Xeon processors and using InfiniBand interconnect, came in number six, with a sustained rating of 38.3 teraflops on the Linpack test (this machine is noteworthy for how inefficient it is, at least on Linpack). Sandia is also home to number nine on the list, the "Red Storm" Linux-Opteron massively parallel cluster built by Cray, which uses 10,880 2 GHz single-core Opterons and which delivers 36.2 teraflops of number-crunching power. Sun Microsystems, after a long absence anywhere near the top of the list, has broken into the Top 10 with the "Tsubame" cluster it has built for the Tokyo Institute of Technology in conjunction with NEC. This machine, which uses the future eight-core "Galaxy" servers and math co-processors from ClearSpeed Technology, has 10,368 processor cores and a sustained Linpack rating of 38.2 teraflops. Forschungszentrum Juelich in Germany has installed a Blue Gene machine, too, which is rated at 37.3 teraflops and which comes in at number eight on the Top 500 list. Rounding out the Top 10 is Earth Simulator, which was built by NEC for the Japanese government using its SX6 vector processors--5,120 of them, in fact--as elements in a parallel system that has a sustained rating of 35.9 teraflops. This machine dates from 2002, making it one of the oldest boxes on the list. It is amazing that it is still in the Top 10, in fact.

In terms of architecture, clusters, by which is meant machines linked together using InfiniBand, Quadrics, Myrinet, or Gigabit Ethernet connections and the MPI clustering software, account for 364 of the 500 systems on the list, or 72.8 percent of machines and about 50.7 percent of aggregate floating point power in the list. Massively parallel processors, or MPPs, which have more tightly coupled interconnections and, account for 98 machines, or about 19.6 percent of the systems in the list, but 44 percent of the aggregate power on the Top 500 list of supers. The remaining supers use a constellation architecture and make up a tiny portion of the list in terms of machines, flops, or CPUs. The Top 500 list encompasses machines that have a total of 873,595 processors. Intel's 32-bit Xeon chips are used in 144 machines (28.8 percent of the list), while the 64-bit Xeons are used in 118 machines (23.6 percent). Itanium processors are used in 37 machines (7.4 percent), while AMD's single-core Opterons are used in 80 machines (16 percent). IBM's Power processors are used in 83 machines, or 16.6 percent of the boxes, but because the Blue Gene boxes have so many of them (as well as other Power-based supers in the list), Power processors make up 43.2 percent of total processors in the Top 500. Other RISC architectures are not really worth going into in any detail. They will soon drop from the list if trends persist.

By operating system, Linux remains the most popular platform, by far, thanks largely to its development in academia in the late 1990s as a operating system suitable for clusters and competing against RISC/Unix servers and their MPP architectures. Linux is used on 391 machines, while Windows Server 2003 is used on two boxes. The combined Windows boxes account for 10.5 teraflops of computing power, or a fraction of a percent of the aggregate computing on the list. It's a start, and Windows Compute Cluster Server 2003 is about doing to Linux in the supercomputer business what Linux did to RISC/Unix architectures. MPI is MPI, and it will be interesting to see how this plays out. Unix made up the remaining 107 systems on the list.

By vendor, IBM is still the king of the Top 500 list, with 240 machines and 2.3 petaflops of peak computing capacity on the list. Hewlett-Packard, which is the revenue market share leader in the overall HPC server market and has been for years, had 157 systems on the list accounting for 850.9 teraflops of peak computing capacity. Cray has 16 machines on the list, with a total of 188.7 teraflops of capacity, Dell has 22 systems with 211.5 teraflops of capacity, and SGI has 12 machines on the list for a total of 136.6 teraflops. There are another two dozen vendors on the list, and six homemade machines.

By geographical region, the United States has the most machines on the list, with 298 supers, or 60 percent of the list. Europe has 83 machines, which is a drop from the 100 systems it had on the Top 500 list only six months ago. Germany is losing ground, with only 17 machines, compared to the United Kingdom, which is gaining ground and now has 35 systems. Countries in the Asia/Pacific region accounted for 93 machines on the list, up from 66 machines in the November 2005 ranking. China has 28 of those machines, by the way, almost equal with the 29 in Japan.

The way Moore's Law is going, sometime around June 2008, someone is going to break the 1 petaflops barrier and the Top 500 list should easily break an aggregate 10 petaflops of computing capacity. The question is: What on earth will we do with all of this capacity?

RELATED STORIES

Linux Clusters Continue to Expand in Top 500 Supers Ranking

Top 500 Supers List Dominated By Exotic Clusters

Top 500 Supers List Dominated by Teraflops-Class Machines

Linux, X86 Clusters Take Over Top 500 Supercomputer Ranking

Top 500 Supercomputer Ranking Gets Top Heavy