Volume 18, Number 42 -- November 30, 2009

How Does 800,000 CPWs in a 2U Server Grab You?

Published: November 30, 2009

by Timothy Prickett Morgan

At the SC09 supercomputing trade show in Portland, Oregon, last week, I got a tour of the very first Power7-based server that IBM is talking publicly about. It is called the Power7 IH supercomputer node, and it is one of the 2,048 nodes in the "Blue Waters" massively parallel machine that Big Blue is building for the University of Illinois.

There is not a lot of i5/OS angle to this machine, I will admit, but some of the nifty chip and networking technologies that IBM is deploying in Blue Waters can--and probably will--be deployed in commercial servers running AIX, Linux, and i operating systems.

As I explained back in September, IBM is taking four eight-core Power7 chips and integrating them on a single piece of ceramic to create a monster chip with 32 cores and 1 teraflop of aggregate number-crunching power running at 4 GHz. In the picture to the right, the chip package on the left is this multichip module. This MCM package has 512 GB/sec of aggregate memory bandwidth and 192 GB/sec of I/O bandwidth. Each MCM has 16 DDR3 memory modules allocated to it, according to Alan Brenner, a senior technical staff member of the server and network architecture team within IBM's Systems and Technology Group who walked me through this monster node. That is 4 GB per core. IBM is able to cram a 16-socket drawer (eight processor MCMs, eight switches, and 1 TB of main memory) into a space that measures 2U high (just like most workhorse servers, which are 3.5 inches), but which is 39 inches wide (not 19 inches like normal servers) and is 72 inches deep.

The Power7 multichip module (left) and IH node hub/switch (right).

The other neat part of the Blue Waters machine is a hub/switch module, which is next to the Power7 MCM in the picture above. This hub/switch fits into the same-sized socket as this Power7 IH processor MCM (they both have 5,336 pins). This hub/switch module takes the best ideas from IBM's "Federation" supercomputing switches (which were used in the ASCI Blue and ASCI Purple supers) and InfiniBand, and mixes them up to create a new, flat switch interconnect that can link 2,048 drawers together into a reasonably flat and low-latency network. The hub/switch provides connectivity between the right Power7 nodes on the board through the motherboard underneath it all, and provides optical links out to adjacent drawers, which are clustered into four-drawer units called supernodes. Blue Waters is 512 of these supernodes linked together.

Take a look at the Power7 IH drawer, which weighs just over 300 pounds fully loaded and has 16 PCI-Express 2.0 x16 slots off to the left. Power supplies and water-cooling links are on the right, with memory banks wrapping around the processors, and the hub/switches sandwiched in between the left memory bank and the I/O slots.

The Power7 IH "Blue Waters" server.

While I like processing as much as anybody, it was the hub/switch that made me stop and think. Brenner explained that the hub/switch module delivered a total of 1,128 GB/sec of aggregate bandwidth. The host connection between the Power7 MCMs inside a single drawer is rated at 192 GB/sec, with another 336 GB/sec of connectivity to the seven other local nodes on the drawer. There is also 240 GB/sec of bandwidth between the nodes in a four-drawer supernode, and 320 GB/sec dedicated to linking nodes to remote nodes in the entire Blue Waters machine. And because there needs to be a way to talk to disks and such, IBM tossed in another 40 GB/sec of general purpose I/O bandwidth.

Imagine taking a single drawer of this box and using the hub/switch as the backbone of a SMP/NUMA server. You'd have 1 TB of memory and around 800,000 CPWs of aggregate transactional processing capacity. Cut the memory in half and slap in some flash storage to interface with external disk storage that links in through the substantial bandwidth leading out of the drawer, and you could create an incredibly powerful transaction processing engine.

I can imagine one of those Power7 MCMs replacing an entire Power 570 box using Power6+ chips. It might fit into a 1U server with lots of room to spare and deliver around 100,000 CPWs of AS/400-style power.


Rolling Thunder Rollout for Power7 Processors Next Year

Steady as She Goes for IBM's Third Quarter

Start Planning for Power7 Iron Now

The Feeds and Guessed Speeds of Power7

IBM to Reveal Power7 Secrets at Hot Chips

Power 7: Lots of Cores, Lots of Threads

IBM Touts Power Systems Prowess on SAP Tests

With No Power6 QCMs, IBM Waits for Power7

IBM Launches Power6+ Servers--Again

Come On Out, Power6+, You Win

Power vs. Nehalem: Time to Double Up and Double Down

Power vs. Nehalem: Scalability Is So 1995, Cash is So 2009

IBM and Resellers Do the iLoyalty Blitz

i Roadmaps: Here Be Dragons

IBM Doubles the Cores on Midrange Power Systems

More Power7 Details Emerge, Thanks to Blue Waters Super

Bang for the Buck: Raising the System iQ

                     Post this story to
               Post this story to Digg
    Post this story to Slashdot

Sponsored By

Value-Based DR Lies in the Space
Between Tape and HA

                                                             · SaaS (VAULT400)
                                                             · Appliance
                                                             · Licensing
                                                             · High Availability

Solutions that work for you!

Read our latest white paper and learn how we can assist you
in your backups/restores and DR program.

Editor: Timothy Prickett Morgan
Contributing Editors: Dan Burger, Joe Hertvik, Brian Kelly, Shannon O'Donnell,
Mary Lou Roberts, Victor Rozek, Kevin Vandever, Hesh Wiener, Alex Woodie
Publisher and Advertising Director: Jenny Thomas
Advertising Sales Representative: Kim Reed
Contact the Editors: To contact anyone on the IT Jungle Team
Go to our contacts page and send us a message.

Sponsored Links

Profound Logic Software:  FREE OnDemand Webinar. Learn how to easily build and extend i apps
LANSA:  Take your apps to a new dimension with RAMP. FREE Webinar!
Manta Technologies:  Your complete source for IBM i training



IT Jungle Store Top Book Picks

Easy Steps to Internet Programming for AS/400, iSeries, and System i: List Price, $49.95
The iSeries Express Web Implementer's Guide: List Price, $49.95
The System i RPG & RPG IV Tutorial and Lab Exercises: List Price, $59.95
The System i Pocket RPG & RPG IV Guide: List Price, $69.95
The iSeries Pocket Database Guide: List Price, $59.00
The iSeries Pocket SQL Guide: List Price, $59.00
The iSeries Pocket Query Guide: List Price, $49.00
The iSeries Pocket WebFacing Primer: List Price, $39.00
Migrating to WebSphere Express for iSeries: List Price, $49.00
Getting Started With WebSphere Development Studio Client for iSeries: List Price, $89.00
Getting Started with WebSphere Express for iSeries: List Price, $49.00
Can the AS/400 Survive IBM?: List Price, $49.00
Chip Wars: List Price, $29.95

Four Hundred Stuff
Security Vendors Take Aim at HITECH Act for Healthcare IT Overhaul

Raz-Lee Adds Object-Level Security to i OS Security Suite

i365 Plans Multi-Platform Backup and Recovery Based on Windows Server

Data Masking Tool from dataguise to Get DB2/400 Support

Kronos Unveils Software, New Sales Approach at Conference

Four Hundred Guru
File Caching in RSE

Message Received, But Not Understood

More with the WDSc Tasks View

Four Hundred Monitor
Four Hundred Monitor's
Full iSeries Events Calendar

System i PTF Guide
November 21, 2009: Volume 11, Number 47

November 14, 2009: Volume 11, Number 46

November 7, 2009: Volume 11, Number 45

October 31, 2009: Volume 11, Number 44

October 24, 2009: Volume 11, Number 43

October 17, 2009: Volume 11, Number 42

October 10, 2009: Volume 11, Number 41

TPM at The Register
Fujitsu gung-ho on eight-core 'Venus' Sparc

IBM shows off Power7 HPC monster

Super Micro primes 'Magny-Cours' Opterons

Big Blue murders Cell blade servers

How to network at a supercomputing show

HP takes one in the servers

Cray previews XT6 Opteron nodes

PC sales bounce up (and down)

IBM chases HP (and Sun) with tiny mem prices

Nvidia previews next-gen Fermi GPUs

Al Gore entertains the supercomputer troops

IBM squishes systems software into new business unit


Infinite Software
Bytware StandGuard Security
Maximum Availability

Printer Friendly Version

IBM Slashes Power Systems Memory Prices

A New Look for the COMMON Session Grid

SSD Performance: Be Careful Before You Buy

Mad Dog 21/21: The Fox in IBM's Storage Henhouse

How Does 800,000 CPWs in a 2U Server Grab You?

But Wait, There's More:

Reader Feedback on IBM Smart Business Moves into Italy . . . IBM Pushes Smarter Mid-Market IT Projects with More Financing . . . SaaS Sales Up Smartly Despite (or Because Of) the Economy . . . Math, Science, and Engineering: A Better Career These Days? . . . AMD Taps IBM Chiphead for Board of Directors . . .

The Four Hundred


Subscription Information:
You can unsubscribe, change your email address, or sign up for any of IT Jungle's free e-newsletters through our Web site at

Copyright © 1996-2009 Guild Companies, Inc. All Rights Reserved.
Guild Companies, Inc., 50 Park Terrace East, Suite 8F, New York, NY 10034

Privacy Statement