SUNY Buffalo Builds 1.3 Teraflops Linux-Blade Super

by Timothy Prickett Morgan

In what will very likely become a growing trend, the State University of New York has deployed a new parallel supercomputer comprised entirely of clustered blade servers rather than clustered rack-mounted servers. The blade super will be installed at the SUNY Buffalo campus, which has a supercomputer center dedicated to science, visualization, and bioinformatics, and will be used to study the structure of proteins in the hope of treating diseases and discovering new drugs.

The blade super that IBM is building for SUNY Buffalo is comprised of 266 two-way HS20 Xeon DP-based BladeCenter blade servers, each equipped with 2.8 GHz processors and 1 GB of main memory. The configuration also includes seven xSeries 345 two-way Xeon DP servers and 5 TB of FAStT700 storage arrays that act as a data repository that the blade super accesses as it performs its protein simulations. The IBM configuration will be rated at approximately 1.3 teraflops and it will be running Red Hat Advanced Server 2.1 (not the new Linux 3.0, probably because of application compatibility issues).

In early 2002, SUNY Buffalo had contracted with Dell to build a giant 2,000-node Linux cluster running Red Hat and Platform Computing's LSF grid middleware, and this 5 teraflops machine is still the main machine that the bioinformatics center uses today. (A different SUNY Buffalo computing facility bought a 300-node Dell Linux cluster in the fall of 2002.)

According to Jeffrey Skolnick, director of the center of excellence in bioinformatics at SUNY Buffalo, he was looking to expand that cluster, which he reckons has another year of useful life in it. He called back Dell and also brought in IBM, Hewlett-Packard, and LinuxNetworx. Skolnick speaks bluntly, and he only really cares about computers in as much as they allow the university to do research. "I just want to do my science, and I don't want to have to think about it." He said that in the bidding process, Dell seemed less enthusiastic than it had been. He said that he was not happy with the level of technical expertise and support that Dell had given him over the past couple of years, and went with IBM for the new blade super not just because of the much smaller form factor, but because of the support and technical expertise (often though IBM Research) that Big Blue brought to bear to win this latest deal.

That BladeCenter cluster goes live on April 1, and in about a year's time a very big computing deal will be in play at the SUNY Buffalo as Skolnick retires that 5 teraflops cluster. And whoever wins that deal is going to be on the inside track for a very large potential deal, since Skolnick says that he has aspirations to build a 25,000-processor cluster within a few years. "People said I was crazy when I said I wanted 4,000 processors, and they are saying the same thing today," Skolnick said laughing last week when interviewed. "But all I need to do now is get together the money."

Skolnick won't divulge what he paid for the IBM blade cluster, but a little estimating suggests that he would not need very much money to build such a machine today--at least compared to a few years ago. A top-of-the-line HS20 with 3.2 GHz/2 MB Xeon DPs and 2 GB of main memory costs a little over $7,000 in a single-unit quantity. So buying 266 of these would be about $1.8 million; the storage might be a couple of hundred thousand bucks since midrange storage costs around $20 to $35 per GB, depending on the features. With discounts, it is probably around $1.5 million for the hardware. At today's prices, using the best blades IBM sells, that 12,500-node cluster Skolnick is dreaming about would cost only around $90 million and would be rated at around 60 teraflops. Two years now, such a machine will only require half as many nodes to deliver the same power and will probably cost half as much--maybe even less.