IBM Opens Blue Gene/L Utility Center in Minnesota

by Timothy Prickett Morgan

It is perhaps fitting that the new supercomputing on demand center that was opened up by IBM last week in Minnesota, home of Control Data Corp, the place where Seymour Cray learned to build supercomputers and the place where he eventually founded another supercomputer company that bore his name. You might think that supercomputer companies pick Minnesota because the HVAC bills are lower, and maybe they do. For whatever reason, Minnesota has always been a hotbed of supercomputing, and Big Blue is continuing in that tradition by opening its first Blue Gene utility computing center in its Rochester, Minnesota, facilities.

The Rochester labs are where IBM's minicomputers, 3.5-inch disk drives, and early 64-bit PowerPC processors were all created. It is also where IBM manufactures its Power-based Linux, AIX, and OS/400 servers these days, and where a lot of the design work in its Power servers gets done. Suffice it to say, IBM Rochester has no shortage of smart nerds, and that is why the Blue Gene/L supercomputer is built there and that is also why, says Mark Solomon, the man behind the supercomputer utility centers at IBM, the company decided to locate the first Blue Gene utility in Rochester.

Technically, the facility is called the Deep Computing Capacity on Demand Center, and it is starting out modestly if you are looking at rack counts, but pretty impressively if you count teraflops. The center will have a single rack of the eServer Blue Gene servers. That doesn't sound like so much, right? Well, each rack has 1,024 dual-core processors running at 700 MHz (based on a stripped down 32-bit PowerPC 440 embedded processor), and that rack delivers a 5.7 teraflops of processing power. That is a lot of computing power to cram in one space, which is why IBM designed Blue Gene in the first place. IBM's first customer for Blue Gene is the U.S. Department of Energy, which is installing a 64-rack, 360-teraflops Blue Gene/L system at Lawrence Livermore National Laboratory to help it simulate and manage the country's stockpile of nuclear weapons.

The Blue Gene/L machines run a stripped down Linux kernel created by IBM Research on its processing nodes (that's what the "/L" stands for in Blue Gene) and run Novell's SuSE Linux Enterprise Server 9 on I/O and management nodes in the cluster. IBM's list price for a rack of Blue Gene/L servers is $2 million, including the cost of Linux licenses but any external disk storage. At that price, Blue Gene/L is about an order of magnitude less expensive (just under 30 cents per flops) than parallel supercomputer clusters based on RISC/Unix servers and high-speed switch interconnects.

Back in March 2004, IBM was showing off a prototype deskside Blue Gene/L machine that had 64 Blue Gene server nodes (128 cores) in a deskside box, and the company was suggesting that this machine would be sold to developers to give them a chance to play around with the new Blue Gene architecture. It is unclear if IBM is pushing ahead with the commercialization of Blue Gene in this manner, but the new Rochester utility is also clearly being positioned as a means for developers to test their code on a much more powerful Blue Gene box without having to acquire anything but time on the Blue Gene/L utility. IBM says it has created a highly secure virtual private network to give customers access to the utility, which is particularly important since the kinds of military and commercial simulations that run on supercomputers are by their very nature something that organizations want to keep secret.

Even with the impressive price/performance of the commercialized Blue Gene servers, a couple of million bucks is still a lot to pony up for what is still largely an experimental supercomputer. While IBM has been pushing Power and X86 clusters in its On Demand centers in Poughkeepsie, New York, in Houston, Texas, and in Montpelier, France (which together have over 5,200 processors available for utility-style use by commercial customers), these architectures are well known and therefore customers know what they are buying into.

"Blue Gene is a new and exotic technology, and On Demand gives people a reasonable way to get going," explains Solomon. He says that IBM is making a 32-node partition (just under 180 gigaflops) the minimum piece of the utility that customers can buy, and they have to buy it for at least a week. After that, they can buy capacity in daily increments. By the way, he says that in the Blue Gene's architecture, the toroidal interconnect doesn't kick in until 512 nodes are clustered together, which means all of the nodes in the rack are linked using a flatter hierarchical scheme. Over time, perhaps by the end of the year depending on customer demand, IBM will expand the Blue Gene utility to four racks and the toroidal interconnect can come into play.

Solomon said he was not at liberty to say what the computing capacity cost is on the Blue Gene utility, except to say that it is "significantly lower" than for on-demand capacity for X86 clusters. Other sources at IBM tell me that the company is charging $10,000 per week for that 180 gigaflops slice, which works out to 33 cents per gigaflops per hour. While buying a whole rack of Blue Gene and amortizing the cost of raw capacity over three years (not including HVAC, electricity, or administration costs) brings in a price of about 1.3 cents per gigaflops per hour, the price that IBM is charging for a rental slice of Blue Gene is nonetheless pretty attractive for companies who do not want to build their own HPC data centers and who just want to be experts at writing HPC software and crunching numbers.

And while that cost savings is important on the front-end of the deal, Solomon says that after customers use these utilities, they realize something else is more important. "Customers figure out that they can do things they could not do before," he says. "Being able to have ten times as much computing capacity, even for a short time, allows companies to compete on a level that they could not compete at before. This is the real reason customers love this." In fact, IBM didn't have to go looking for its first Blue Gene utility customer. QuantumBio, a provider of drug discovery software and services for big pharmaceutical companies based in State College, Pennsylvania (and loosely affiliated with Penn State), came to IBM because of the Blue Gene architecture and said it wanted to buy utility capacity on the box.