Beep, Beep: Roadrunner Linux Super Breaks the Petaflops Barrier

Published: June 17, 2008

by Timothy Prickett Morgan

This week, the International Supercomputing Conference 2008, one of the two big events dedicated to high performance computing, is underway with week in Dresden, Germany, and the star of that show will be a massively parallel, hybrid supercomputer called Roadrunner, created by IBM for Los Alamos National Laboratory, that has officially broken through the 1 petaflops peak performance barrier. Beep, beep!

One petaflops is equal to 1,000 teraflops, which is in turn equal to 1,000 billion floating point operations per second. It has taken a long time to reach this much-sought goal. The first commercial supercomputer to break through the 1 teraflops barrier was arguably--and I say arguably because people argue about this--was probably the Cray T3E-900, which had 2,048 processors when it shipped in November 1996 and was rated at 1.8 teraflops. Intel's ASCI Red, which was a custom-built super for Sandia National Laboratory, was rated at around 1.8 teraflops when it was installed in 1997, and it was the largest machine, in terms of Linpack Fortran benchmark peak performance, for a number of years before massively parallel Unix and Linux machines went mainstream. Now, several dozens of teraflops is no big deal, but a decade ago, a petaflops of number-crunching power seemed like a dream.

IBM, Cray, Sun Microsystems, Silicon Graphics, and a number of other vendors are all reaching to break the petaflops barrier, and push as hard as possible toward 10 petaflops and beyond. IBM has two different designs that it hopes to push up beyond 1 petaflops, the Roadrunner Opteron-Cell hybrid, which is expected to eventually reach 1.6 petaflops, and the BlueGene/P massively parallel PowerPC machine, which is going into the Argonne National Laboratory. The U.S. Department of Energy is ponying up the cash for both of these machines--and many others--in an effort to get enough computing capacity to redesign nuclear warheads and test that older warheads will still explode correctly on behalf of the military. No one is putting a public price tag on these supers, but the DOE has spent billions of dollars over the past decade to upgrade the gear in the national labs it runs. According to a report in the New York Times pegged the cost of the Roadrunner machine at $133 million.

The Roadrunner machine is based on IBM servers that had a total of 6,948 dual-core Rev F Opteron processors (for a total of 13,896 cores) that are coupled to 12,690 of a second-generation of the Cell Power processor that Big Blue created in conjunction with Sony and Toshiba. (Sony uses the Cell chips, which have a 64-bit Power core and eight auxiliary processors that can be used to render graphics or do mathematical calculations, in the PlayStation game console.) The Cell designs brings 101,520 auxiliary math units, plus another 12,690 Power cores, to bear to assist the applications that are running on the Opteron processors, which obviously come from Advanced Micro Devices. The Opteron chips were originally supposed to be packed into System x3775 rack servers, according to what IBM told us back in September 2006, but instead Roadrunner uses a completed bladed design and is using dual-core LS21 Opteron-based blades.

For every one Opteron blade, there are two QS22 Cell-based blade servers, which were just announced a few weeks ago. The QS22 blade has two Cell chips on it, and provides 217 gigaflops of double-precision floating point processing power and 460 gigaflops of single precision performance. The QS22 blade has InfiniBand daughter cards as well as two Gigabit Ethernet ports and supports up to 32 GB of 800 MHz DDR2 main memory and larger I/O buffers to keep the Ethernet and InfiniBand links from flooding the processors. All three of these blades are linked together with a custom-built I/O expansion blade, which uses PCI-Express x8 links to connect the QS22 blades to the LS21 blades (where applications are actually running) and which links the blades behind it to other so-called "triblade" configurations and to shared storage arrays through InfiniBand links. Voltaire was tapped to provide 26 of its 2102 InfiniBand switches to link all of the blades together and to their storage; each switch has 288 ports that provide 20 Gb/sec of bandwidth. The cluster does have a dozen System x 3755 servers running its file system.

Roadrunner is a Linux machine, of course, but which one is running on the hybrid cluster is not known yet. Novell last week was bragging about having 20 out of the top 50 supercomputers in the world running SUSE Linux Enterprise Server and did not mention Roadrunner--and for good reason. IBM certified only Red Hat Enterprise Linux 5 on the Cell blades, and indeed, RHEL 5 is running across the entire Roadrunner hybrid cluster.

We will, of course, provide you coverage of the happenings and announcements at ISC 2008, including the Top 500 supercomputer rankings, in next week's issue.

RELATED STORIES

IBM to Break Petaflops Barrier with Blue Gene/P

IBM to Build 1.6 Petaflops Super for Los Alamos Lab

Cray Lands $200 Million Linux-Opteron Super Deal with DOE

"Cell" PowerPC Partners Try to Round Up Support for the Chip

IBM, Sony, and Toshiba Lift Curtain on Cell Chip

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot