Volume 4, Number 36 -- October 2, 2007

HP Wins 163 Teraflops Opteron Super Deal at PNNL

Published: October 2, 2007

by Timothy Prickett Morgan

Hewlett-Packard has not been near the top of the global list of supercomputer sites since the ASCI-Q Unix supercomputer built by Compaq's Alpha RISC servers and its Tru64 Unix were riding high at Los Alamos National Laboratory back before HP even bought Compaq. But this week, Pacific Northwest National Laboratory is going to vault HP into the upper echelons of the Top 500 list by installing a 163 teraflops Opteron-based Linux cluster that is being built by HP.

The new machine, which has not been given a nickname yet but which was requisitioned under the procurement name HPCS-3, will be comprised of 2,310 two-socket ProLiant servers using Advanced Micro Devices' just-shipping "Barcelona" quad-core Opteron Rev F processors. PNNL has one of the largest Itanium-based systems in the world--a machine called MPP2 that has 1,936 single-core 1.5 GHz Itanium 2 processors, based on two-socket Integrity rx2600 servers and interconnected with a Quadrics network. This machine is rated at a peak performance of 11.6 teraflops, and runs a modified version of Linux.

The new HPC cluster deal is good news for AMD, which has been hurt badly by Intel's successful quad-core "Clovertown" Xeon 5300 processors, which plug into two-socket servers and which were announced last November, 10 months ahead of production shipments of Barcelona. That PNNL did not upgrade to "Montecito" dual-core Itanium 9000 processors is a public relations loss for Intel, and that PNNL opted for Barcelona instead of Clovertown is also some bad news. No matter what, HP is the winner, since it sells two-socket machines based on Itanium, Opteron, and Xeon processors.

HP sure was not going to give out any reason behind the shift from Itanium to Opteron. "HPC customers make their server and architecture choices at a particular time," explains Scott Misage, director of advanced development at HP's high performance computing division. "PNNL made a choice based on a point-in-time analysis using its own codes, and this time Opteron-based machines won. We continue to believe that Itanium is interesting and useful for HPC workloads," Misage added. In particular, Itanium-based fat-node servers are often popular in clusters that need a lot of memory bandwidth inside a node to support their workloads.

While the Itanium processors have excellent floating point performance, they run a bit hot, they are relatively expensive, and they only come in low-volume and somewhat more expensive systems. And hence, academic and government research centers, which are among the stingiest server buyers in the world because they have to buy so many servers to run their code, do not want to pay an extra dime for a flops of computing power, and do not generally want to invest in Itanium-based nodes for their clusters.

PNNL runs a set of homegrown applications for computational chemistry and molecular biology called NWChem. The lab is one of a handful of big research institutions funded by the U.S. Department of Energy to do research in physics and chemistry. NWChem runs on Sparc, Power, Alpha, MIPS, and X86-X64 platforms running variants of Unix or Linux. To do more complex modeling, PNNL needs to substantially increase its performance, which is what the HPCS-3 requisition is all about.

The HPCS-3 machine will be installed in two phases at PNNL, according to Misage, with 603 ProLiant machines going in during January 2008 and the remaining 1,746 machines going in by September 2008. The machines will be networked together using InfiniBand 4x interconnect (in a fat tree topology) and will link to a Lustre-based cluster file system created by HP called the Scalable File Share, which in this case will have over 21,000 disk drives and over 250 TB of capacity. The server nodes in the cluster will have a total of 37 TB of main memory, and will use Barcelona Opteron cores running at 2.2 GHz. The 18,840 cores will deliver about 163 teraflops of peak performance, and somewhat less than that (maybe 120 teraflops by my estimate) of sustained performance as measured by the Linpack Fortran HPC benchmark test. That peak performance will put HP in the top 10 of supercomputer sites worldwide, and it could rank as high as number two on the list, after the Blue Gene/L Linux box at Lawrence Livermore National Lab, depending on how other HPC centers do their upgrades in the next year.

HP is building and assembling the two stages of the HPCS-3 machine in its Houston, Texas, factory, integrating the hardware and software there, and then breaking it down and shipping it up to Richland, Washington.

PNNL has also just bought a 128-thread XMT system from supercomputer rival Cray, which last year retrofitted its MTA-2 massively multithreaded Tera Computer processor to slip into an Opteron Rev F socket so it could plug into "Red Storm" Opteron-Linux superclusters, now commercialized as the Cray XT4. The Cray XMT systems can have over 128 TB of main memory, over 8,000 processors, and more than 1 million processor threads available to workloads. Such a machine, were it to be built, would make an excellent pattern recognition machine. Which is probably why the U.S. government has some much bigger XMT systems running somewhere.


AMD Gets Aggressive About Watts with Quad-Core Barcelonas

Top 500 Supers: Moore's Law Is Alive and Well

Dual-Core Processors Begin Takeover of Top 500 Super Ranking

Top 500 Supers: Brace Yourself for Petaflops Systems

HP, Red Hat Launch Sophisticated File Systems for Linux

Lintel Iron Makes Headway in the Top 500 Super Rankings

                     Post this story to
               Post this story to Digg
    Post this story to Slashdot

Sponsored By

CanIt-PRO Anti-Spam Software for Linux/UNIX

· Flexible spam and virus blocking
· Unparalleled end-user control
· Access to source code? Of course!

Stop spam at the mail server on YOUR terms with CanIt-PRO from Roaring Penguin Software. Combines the power of MIMEDefang and SpamAssassin™ to provide unparalled end-user control and an easy-to-use web interface.

Download free evaluation software at

Editor: Timothy Prickett Morgan
Contributing Editors: Dan Burger, Joe Hertvik, Kevin Vandever,
Shannon O'Donnell, Victor Rozek, Hesh Wiener, Alex Woodie
Publisher and Advertising Director: Jenny Thomas
Advertising Sales Representative: Kim Reed
Contact the Editors: To contact anyone on the IT Jungle Team
Go to our contacts page and send us a message.

Sponsored Links

COMMON:  Join us at the annual 2008 conference, March 30 - April 3, in Nashville, Tennessee
Vibrant Technologies:  Quality Used Servers, Storage & Networking Hardware at up to 80% off new
Bytware:  The power of McAfee with award-winning StandGuard Anti-Virus for Linux


IT Jungle Store Top Book Picks

The System i Pocket RPG & RPG IV Guide: List Price, $69.95
The iSeries Pocket Database Guide: List Price, $59.00
The iSeries Pocket Developers' Guide: List Price, $59.00
The iSeries Pocket SQL Guide: List Price, $59.00
The iSeries Pocket Query Guide: List Price, $49.00
The iSeries Pocket WebFacing Primer: List Price, $39.00
Migrating to WebSphere Express for iSeries: List Price, $49.00
iSeries Express Web Implementer's Guide: List Price, $59.00
Getting Started with WebSphere Development Studio for iSeries: List Price, $79.95
Getting Started With WebSphere Development Studio Client for iSeries: List Price, $89.00
Getting Started with WebSphere Express for iSeries: List Price, $49.00
WebFacing Application Design and Development Guide: List Price, $55.00
Can the AS/400 Survive IBM?: List Price, $49.00
The All-Everything Machine: List Price, $29.95
Chip Wars: List Price, $29.95

The Four Hundred
State of the System i: What's Going On in the Market?

Summit Partners Sells Help/Systems to Audax Group

Reader Feedback on EGL: At Least It's Not Java, But It Ain't RPG, Either

IBM Helps Users Migrate to Power6-Based System i Boxes

Four Hundred Stuff
Boise Cascade Gets 'm-Powered' with mrc Reporting Tool

Red Oak Simulates World's Fastest Human Terminal Operator

Echo Cuts Waste with SPLTOOL Investment

BCD's WebSmart PHP Goes GA

Big Iron
Hosing z/OS.e and Other Withdrawals

Top Mainframe Stories From Around the Web

Chats, Webinars, Seminars, Shows, and Other Happenings

Four Hundred Guru
Weird SQL UNION Results

A Snippet Above the Rest

Admin Alert: What Vendors Want to Know (and How to Get It)

System i PTF Guide
September 29, 2007: Volume 9, Number 39

September 22, 2007: Volume 9, Number 38

September 15, 2007: Volume 9, Number 37

September 8, 2007: Volume 9, Number 36

September 1, 2007: Volume 9, Number 35

August 25, 2007: Volume 9, Number 34

The Windows Observer
Microsoft Launches Business Intelligence Tool for the Masses

Microsoft Revs Betas of Longhorn, Viridian, Vista SP1

IDF Server Wrap Up: Intel to Keep the Pressure on AMD

Sun Ships Intel-Based Galaxy Rack Servers

The Unix Guardian
Sun Enhances Solaris Developer Edition, Adds Support

Sun Ships Intel-Based Galaxy Rack Servers

IDF Server Wrap Up: Intel to Keep the Pressure on AMD

As I See It: Shocking

Four Hundred Monitor
Four Hundred Monitor's
Full iSeries Events Calendar


Roaring Penguin
Storage Guardian

Printer Friendly Version

Red Hat, Reporting Q2, Reorganizes Operations for Growth

An Update from the X64 Server Battlefields

HP Wins 163 Teraflops Opteron Super Deal at PNNL

Sun Ships Intel-Based Galaxy Rack Servers

But Wait, There's More:

What Are IT Employees Up To? Sleeping and Kissing, Apparently . . . Vendors to Squeeze More Juice Out of Linux . . . Sun Buys the Assets of Cluster File Systems . . . NEC and Sun Team for HPC Server Deals in North America . . . Gartner Charts External Disk Array Sales for Q2 . . . IT Shops Consume 2 Million LTO Tape Drives . . .

The Linux Beacon


Subscription Information:
You can unsubscribe, change your email address, or sign up for any of IT Jungle's free e-newsletters through our Web site at

Copyright © 1996-2008 Guild Companies, Inc. All Rights Reserved.
Guild Companies, Inc., 50 Park Terrace East, Suite 8F, New York, NY 10034

Privacy Statement