IBM to Build 1.6 Petaflops Super for Los Alamos Lab
Published: September 12, 2006
by Timothy Prickett Morgan
IBM has snagged a contract with the U.S. Department of Energy to build a 1.6 petaflops hybrid supercomputer for the Los Alamos National Laboratory by the end of 2007 or early 2008. The supercomputer, code-named "Roadrunner" after the New Mexico state bird, is expected to be the fastest supercomputer in the world when it is completed. Roadrunner is also an example of a hybrid processor architecture, because it will be based on both Opteron X64 processors from Advanced Micro Devices and Cell PowerPC processing elements from IBM.
The Opteron, of course, is the 64-bit variant of the X86 processor that has put AMD on the map and Intel on the mat. Supercomputer maker Cray has been peddling Opteron-based supercomputer designs for several years, and in June landed a $200 million contract with Oak Ridge National Laboratory that will see that lab, which is also funded by the DOE, get to the 1 petaflops level by late 2008 or so with a machine nicknamed "Jaguar." This machine is currently rated at 54 teraflops, but could eventually include as many as 24,000 Opteron processor cores. (The exact number will depend on the chips and clock speeds that AMD has available at the time.)
If IBM can pull off the Roadrunner design and deliver it on schedule, IBM will beat Cray to breaking the petaflops barrier. (One petaflops is equal to 1,000 teraflops, which is in turn equal to 1,000 billion floating point operations per second.) The biggest supercomputer in the world today, as measured by raw processing capacity on the Linpack Fortran benchmark test, is IBM's BlueGene/L supercomputer, which is installed at Lawrence Livermore National Laboratory and which is currently rated at 367 teraflops of peak performance. While this machine is very powerful and extremely energy efficient and compact (at least by parallel supercomputing standards), it has an incredible 131,072 dual-core PowerPC processors and that can make it difficult to program and unsuitable for some workloads.
By comparison to these BlueGene PowerPC chips, an Opteron chip with two or four cores has a lot more oomph. And according to Henry Brandt, high performance computing strategy director at IBM, Los Alamos was eager to get started building the foundation of a new supercomputer immediately, so picking an Opteron-based system was one of the obvious choices. IBM pitched a hybrid Opteron-Cell machine because the Cell processor has an incredible amount of number-crunching capability. In fact, the Cell chips make up the bulk of the raw computing capacity in the Roadrunner system.
The Cell chip was developed by IBM in conjunction with Sony and Nintendo, which are deploying Cell chips in their next-generation game consoles, and Toshiba, which is expected to use the Cell chip in various electronic devices such as HDTVs. The Cell chip consists of a 64-bit PowerPC core that has had its out-of-order execution elements removed; it has 64 KB of L1 cache (split evenly between data and instruction caches), 512 MB of on-chip L2 cache, and an integrated Rambus XDR main memory controller. Each Cell also has eight co-processors, which IBM calls synergistic processor units, which are vector math units that are used to boost video and other media processing.
Brandt says that the way that the Opterons will be linked to the Cells has not been precisely determined, but he did say that IBM is not using the new Opteron-based LS42 blade servers for the Opteron portion of the deal. A forthcoming Cell-based blade server, which IBM was prototyping earlier this year, is expected to be launched for IBM's BladeCenter chassis in a matter of weeks, and these will be used in the Roadrunner machine. The Opteron portion of the supercomputer will be comprised of System x3755 servers, which Big Blue announced in early July. These are four-socket boxes, which can therefore support up to eight Opteron cores using the new Rev F Opteron processors, which were announced in late July.
Los Alamos plans to build Roadrunner in three stages. The first stage is to get the Opteron machines installed, which will comprise just under 80 teraflops of computing capacity and 16,000 Opteron processor cores. In the second phase, IBM will provide Los Alamos with some Cell-base blade servers and lots of programming assistance to tune Linux for the Cell chip and, more importantly, for the specific workloads at Los Alamos. IBM won a $35 million contract for this phase of the Los Alamos machine.
Los Alamos, Oak Ridge, Lawrence Livermore National Laboratory, and Sandia National Labs are all funded by the DOE, which is responsible for maintaining the nuclear weapons stockpile for the military. Because the U.S. is no longer allowed to explode nuclear bombs, it has to simulate how these aging weapons behave to make sure they still work as designed.
During that second phase of deployment, which happens during early 2007, experts from IBM Research, Software Group, and Systems and Technology Group will be involved with architecting the machine and tuning it. Work will be done to make compilers automatically vectorize code for the Cell chips.
IBM currently supports the Fedora development release of Linux from Red Hat on the Cell chip, and Los Alamos currently used Red Hat Linux on many of its machines--with custom patches to boost the performance of its applications--on its current machines, and is expected to deploy some variant of Red Hat on the Opteron portion of the Roadrunner cluster.
During the third phase, over 8,000 Cell-based blade servers, with two Cell chips on each blade, will be added to Roadrunner, adding the bulk of the 1.6 peak petaflops of computing power it comprises. Brandt would not say what it is charging for phases two or three of the contract, and he said that IBM was still mulling the way the Opteron and Cell machines will connect to each other. IBM could have added an HTX HyperTransport port to the Cell blade and plugged that into the LS42 blade, which uses HTX ports to link two two-socket blades together to make a four-socket SMP blade. But, for whatever reason, IBM decided to not do this.
The Cell blade uses an IBM-developed bus architecture called the element interconnect bus, or EIB, to link the Cell vector processors to each other and to link Cell chips to each other into processor fabrics, and this interconnect may be incompatible with HyperTransport links, which are used by AMD to link multiple Opterons together. Brandt says that by choosing the four-socket rack-mounted server instead of Opteron-based blades, Los Alamos and IBM can keep their connectivity options between the two types of machines open.
Roadrunner is expected to have a sustained performance of around 1 petaflops on the Linpack benchmark test. It will be enclosed in 360 server racks that take up around 12,000 square feet--about three basketball courts.
Cray Lands $200 Million Linux-Opteron Super Deal with DOE
"Cell" PowerPC Partners Try to Round Up Support for the Chip
IBM, Sony, and Toshiba Lift Curtain on Cell Chip