Volume 20, Number 28 -- August 15, 2011

Resurrect Dead Blue Waters Power7 Supercomputer As IBM iCloud

Published: August 15, 2011

by Timothy Prickett Morgan

You need a supercomputer to predict the future, and unfortunately when you are trying to predict the kind of supercomputer you might need to build the supercomputer to predict that future, you can't have as much knowledge of the future to easily make the predictions. And therefore, sometimes a supercomputer project backed by governments and key IT players gets taken out behind the barn and given the Old Yeller treatment.

Such is the case with the technically impressive "Blue Waters" massively parallel machine that was to be built this year by IBM for the National Center for Supercomputer Applications (NCSA) at the University of Illinois. Blue Waters, which is built using a super-dense packaging of water-cooled Power7 processors, memory modules, and an array of hub/switch chips code-named "Torrent," is a bit ahead of its time in terms of manufacturing, apparently. (I told you all about the Blue Waters server node, formerly known as the Power IH node and launched in July as the Power 775 server, back in November 2009 when I saw them on display at the SC2009 supercomputing show.)

The Power 775 servers can cram 256 Power7 cores and 2 terabytes of main memory to feed them into a server node that is 2U high, 30 inches wide, and six feet deep. Back in November 2009, I estimated that if this Power 775 node was running the IBM i operating system, it would deliver about 800,000 CPWs of performance. I assumed that the clock speed would be a bit lower than on the standard Power 770 and 795 machines and didn't know about the embedded DRAM cache on the Power7 chip then and its dramatic effect on performance, and as it turns out the Power 775 drawer has something more like 1.53 million of aggregate CPWs of IBM i oomph.

The Torrent hub/switch is something you will see more of, albeit probably in a shrunken form and maybe on a chip in some future Power processor. This hub/switch module delivers a total of 1,128 GB/sec of aggregate bandwidth. The host connection between the Power7 multichip modules inside a single drawer is rated at 192 GB/sec, with another 336 GB/sec of connectivity to the seven other local nodes on the drawer. There is also 240 GB/sec of bandwidth between the nodes in a four-drawer supernode, and 320 GB/sec dedicated to linking nodes to remote nodes in the entire Blue Waters machine. And because there needs to be a way to talk to disks and such, IBM tossed in another 40 GB/sec of general purpose I/O bandwidth inside a drawer.

Up to 2,048 of these Power 775 drawers can be linked together into a behemoth using the Torrent interconnect. The Power 775 super has storage and compute drawers, and a balanced configuration with 1,365 compute nodes and 342 storage nodes with 2.7 petabytes of memory and 26.3 petabytes of disk and flash storage. It would cost about $1.5 billion at IBM list price for an aggregate of 349,440 cores that could deliver around 2.1 billion CPWs. To give you some perspective, that is probably about half of the aggregate IBM i processing capacity in the world. That price doesn't include the IBM i software licenses. Even at the modest $2,245 per core that IBM charges for the entry PS700 blade server (with 90 days of Software Maintenance), across those 349,440 cores you would be talking about another $1.5 billion for the operating system software and three years of maintenance. Call it $3 billion for 2 billion CPWs.

And that, it seems, is the problem. That's a fair price for a CPW of oomph, but it is a terrible price for a floating point operation per second (flops). After designing and building three racks of the Blue Waters machines based on the economics it thought it would be dealing with in 2011 way back in 2007, IBM's top brass pulled the plug on the contract. Presumably because IBM could not make money after doing the manufacturing and support job. While supercomputers have been a loss leader for IBM and help drive its overall server technologies--the supercomputer interconnect from a decade ago gets tweaked to be the SMP backplane in tomorrow's machines--Big Blue was not going to take a big loss for Blue Waters, for which it was to be paid $208 million, mostly from the National Science Foundation.

"The University of Illinois and NCSA selected IBM in 2007 as the supercomputer vendor for the Blue Waters project based on projections of future technology development," IBM and NCSA said in a joint statement. "The innovative technology that IBM ultimately developed was more complex and required significantly increased financial and technical support by IBM beyond its original expectations. NCSA and IBM worked closely on various proposals to retain IBM's participation in the project but could not come to a mutually agreed-on plan concerning the path forward."

I talked to John Melchi, head of the administration directorate of the NCSA, and he pinned the killing of Blue Waters on IBM's upper management and said further that as far as NCSA could tell from its early tests on three racks of Power 775s that the machine would work as expected and eventually deliver at least a petaflops of sustained performance.

IBM has to return $30 million to NCSA and the University of Illinois is out some dough it put out to get grad students to tune code for the box. IBM sources say the company is still selling the Power 775 servers, which are available at the end of the month but which are, as you can see, quite expensive.

I think it would be interesting to get IBM i running on these machines and turn it into an IBM iCloud. Tell Sam Palmisano, Big Blue's president, chief executive officer, and chairman, and Tom Rosamilia, general manager of the Power Systems and System z line. I just did. A buck and a half per CPW is a manageable price if you can spread it across enough users. That assumes, of course, that IBM is charging $2,245 per core (an IBM i entitlement on a PS700 core with 90 days of SWMA) instead of $53,000 per core plus $6,000 per year for a year of SWMA on a Power 795. On a balanced Power 775 superserver, that would be $24.8 billion for 2.1 billion CPWs just for the software, plus another $1.5 billion for the hardware. That's a truly stupid price--and it is exactly what Big Blue charges (at list price) for the IBM i software on its largest Power systems iron.

You Young i Professionals want to have some fun? Get IBM to let you build this cloud with the three returned Power 775 racks from the failed Blue Waters super and run it as a business unit inside of IBM Global Services. You can probably rent space from NCSA.


IBM to Ride Growth Waves on Current Iron in 2011

A Little Insight Into the Rest of the Power7 Lineup

How Does 800,000 CPWs in a 2U Server Grab You?

More Power7 Details Emerge, Thanks to Blue Waters Super

                     Post this story to
               Post this story to Digg
    Post this story to Slashdot

Sponsored By

Add audit trails to your master files in minutes with Audit Trail/400 -
without programming or journaling.

Audit Trail/400 answers the questions:
Who changed what? When? How?
What was it before?

Audit Trail/400 simplifies Sarbanes-Oxley requirements,
FDA compliance, debugging, or just general ERP support.

Download a fully-functional,
no-obligation Audit Trail/400 trial from
and have your first audit trails
running in under one hour.

Editor: Timothy Prickett Morgan
Contributing Editors: Dan Burger, Joe Hertvik, Victor Rozek,
Jenny Thomas, Hesh Wiener, Alex Woodie
Publisher and Advertising Director: Jenny Thomas
Advertising Sales Representative: Kim Reed
Contact the Editors: To contact anyone on the IT Jungle Team
Go to our contacts page and send us a message.

Sponsored Links

PowerTech:  2011 Security Event of the Year. September 2223 in Las Vegas. RVSP today!
CNX:  Ready to develop true Web 2.0 applications? Valence Version 3.0 BETA 3 is now available
System i Developer:  Join the Gurus at the RPG & DB2 Summit in St. Louis, October 17-19



IT Jungle Store Top Book Picks

BACK IN STOCK: Easy Steps to Internet Programming for System i: List Price, $49.95

The iSeries Express Web Implementer's Guide: List Price, $49.95
The iSeries Pocket Database Guide: List Price, $59
The iSeries Pocket SQL Guide: List Price, $59
The iSeries Pocket WebFacing Primer: List Price, $39
Migrating to WebSphere Express for iSeries: List Price, $49
Getting Started with WebSphere Express for iSeries: List Price, $49
The All-Everything Operating System: List Price, $35
The Best Joomla! Tutorial Ever!: List Price, $19.95

Four Hundred Stuff
IBM i Server Gets Hip to QR Codes

IBM's Multi-Faceted Mobile Strategy

VAI Gets Social with New Release of S2K Salesforce

Three IBM i Solution Editions Unveiled by Infor, IBM

ClearPointe and EView Pair Up for Remote Management of IBM i Servers

Four Hundred Guru
Adobe Flash Builder for the iSeries Programmer, Part 1

ILE: Decisions, Decisions, Part 2

Admin Alert: An Alternate Way to Port Image Catalogs Between Systems: QFileSrv.400

Four Hundred Monitor
Four Hundred Monitor's
Full iSeries Events Calendar

System i PTF Guide
September 25, 2010: Volume 12, Number 39

September 18, 2010: Volume 12, Number 38

September 11, 2010: Volume 12, Number 37

September 4, 2010: Volume 12, Number 36

August 28, 2010: Volume 12, Number 35

August 21, 2010: Volume 12, Number 34

TPM at The Register
SuperVisor: One hypervisor to virtualize them all

Skytap adds orchestration to dev/test cloud

Adaptive Computing spins up Moab 6.1 control freak

IBM yanks chain on 'Blue Waters' super

Chip makers to strut their stuff at Hot Chips 23

Cray CTO Scott departs

A mixed bag for IT jobs in July

Rackspace: 'OpenStack opens doors, wallets'

New customers drive Teradata to new heights

Amazon virtual private clouds go global

Dell floats Cloud Hadoop clusters

VMware backs down on vSphere 5 virtual memory tax


Linoma Software
WorksRight Software

Printer Friendly Version

The IBM PC Turns 30

Proprietary Machines Prop Up Avnet, Arrow Server Sales

Resurrect Dead Blue Waters Power7 Supercomputer As IBM iCloud

As I See It: Piling On

Temp Workers Remain An IT Hiring Favorite

But Wait, There's More:

Reader Feedback on Lean Mean Green Screens . . . Mertens Takes the Wheel at Sirius as Najim Steps Down . . . WANTED: Unemployed IBM i Professionals . . . IBM Paints X64-Based BladeCenters Sky Blue With Clouds . . . JDA Software App License Sales Stall In Q2 . . .

The Four Hundred


Subscription Information:
You can unsubscribe, change your email address, or sign up for any of IT Jungle's free e-newsletters through our Web site at

Copyright © 1996-2011 Guild Companies, Inc. All Rights Reserved.
Guild Companies, Inc., 50 Park Terrace East, Suite 8F, New York, NY 10034

Privacy Statement