HP Wins 163 Teraflops Opteron Super Deal at PNNL

Published: October 2, 2007

by Timothy Prickett Morgan

Hewlett-Packard has not been near the top of the global list of supercomputer sites since the ASCI-Q Unix supercomputer built by Compaq's Alpha RISC servers and its Tru64 Unix were riding high at Los Alamos National Laboratory back before HP even bought Compaq. But this week, Pacific Northwest National Laboratory is going to vault HP into the upper echelons of the Top 500 list by installing a 163 teraflops Opteron-based Linux cluster that is being built by HP.

The new machine, which has not been given a nickname yet but which was requisitioned under the procurement name HPCS-3, will be comprised of 2,310 two-socket ProLiant servers using Advanced Micro Devices' just-shipping "Barcelona" quad-core Opteron Rev F processors. PNNL has one of the largest Itanium-based systems in the world--a machine called MPP2 that has 1,936 single-core 1.5 GHz Itanium 2 processors, based on two-socket Integrity rx2600 servers and interconnected with a Quadrics network. This machine is rated at a peak performance of 11.6 teraflops, and runs a modified version of Linux.

The new HPC cluster deal is good news for AMD, which has been hurt badly by Intel's successful quad-core "Clovertown" Xeon 5300 processors, which plug into two-socket servers and which were announced last November, 10 months ahead of production shipments of Barcelona. That PNNL did not upgrade to "Montecito" dual-core Itanium 9000 processors is a public relations loss for Intel, and that PNNL opted for Barcelona instead of Clovertown is also some bad news. No matter what, HP is the winner, since it sells two-socket machines based on Itanium, Opteron, and Xeon processors.

HP sure was not going to give out any reason behind the shift from Itanium to Opteron. "HPC customers make their server and architecture choices at a particular time," explains Scott Misage, director of advanced development at HP's high performance computing division. "PNNL made a choice based on a point-in-time analysis using its own codes, and this time Opteron-based machines won. We continue to believe that Itanium is interesting and useful for HPC workloads," Misage added. In particular, Itanium-based fat-node servers are often popular in clusters that need a lot of memory bandwidth inside a node to support their workloads.

While the Itanium processors have excellent floating point performance, they run a bit hot, they are relatively expensive, and they only come in low-volume and somewhat more expensive systems. And hence, academic and government research centers, which are among the stingiest server buyers in the world because they have to buy so many servers to run their code, do not want to pay an extra dime for a flops of computing power, and do not generally want to invest in Itanium-based nodes for their clusters.

PNNL runs a set of homegrown applications for computational chemistry and molecular biology called NWChem. The lab is one of a handful of big research institutions funded by the U.S. Department of Energy to do research in physics and chemistry. NWChem runs on Sparc, Power, Alpha, MIPS, and X86-X64 platforms running variants of Unix or Linux. To do more complex modeling, PNNL needs to substantially increase its performance, which is what the HPCS-3 requisition is all about.

The HPCS-3 machine will be installed in two phases at PNNL, according to Misage, with 603 ProLiant machines going in during January 2008 and the remaining 1,746 machines going in by September 2008. The machines will be networked together using InfiniBand 4x interconnect (in a fat tree topology) and will link to a Lustre-based cluster file system created by HP called the Scalable File Share, which in this case will have over 21,000 disk drives and over 250 TB of capacity. The server nodes in the cluster will have a total of 37 TB of main memory, and will use Barcelona Opteron cores running at 2.2 GHz. The 18,840 cores will deliver about 163 teraflops of peak performance, and somewhat less than that (maybe 120 teraflops by my estimate) of sustained performance as measured by the Linpack Fortran HPC benchmark test. That peak performance will put HP in the top 10 of supercomputer sites worldwide, and it could rank as high as number two on the list, after the Blue Gene/L Linux box at Lawrence Livermore National Lab, depending on how other HPC centers do their upgrades in the next year.

HP is building and assembling the two stages of the HPCS-3 machine in its Houston, Texas, factory, integrating the hardware and software there, and then breaking it down and shipping it up to Richland, Washington.

PNNL has also just bought a 128-thread XMT system from supercomputer rival Cray, which last year retrofitted its MTA-2 massively multithreaded Tera Computer processor to slip into an Opteron Rev F socket so it could plug into "Red Storm" Opteron-Linux superclusters, now commercialized as the Cray XT4. The Cray XMT systems can have over 128 TB of main memory, over 8,000 processors, and more than 1 million processor threads available to workloads. Such a machine, were it to be built, would make an excellent pattern recognition machine. Which is probably why the U.S. government has some much bigger XMT systems running somewhere.

RELATED STORIES

AMD Gets Aggressive About Watts with Quad-Core Barcelonas

Top 500 Supers: Moore's Law Is Alive and Well

Dual-Core Processors Begin Takeover of Top 500 Super Ranking

Top 500 Supers: Brace Yourself for Petaflops Systems

HP, Red Hat Launch Sophisticated File Systems for Linux

Lintel Iron Makes Headway in the Top 500 Super Rankings

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot