Terra Soft to Build Cell-Based Super Out of PS3 Beta Iron

Published: October 10, 2006

by Timothy Prickett Morgan

Commercial Linux distributor Terra Soft Solutions is best known for its PowerPC and Power Linux variant. But now it will be known for another reason. Terra Soft has been tapped by Sony's Computer Entertainment division to make good use of the beta versions of the PlayStation 3 machines that were formerly in the hands of the developers of games for those future Power-based consoles. As many of us have been joking might happen, Terra Soft is going to build two supercomputer clusters based on the machines.

Terra Soft made the PS3 cluster announcement today, and provided some detail on the "E.coli" and "Amoeba" clusters it will build in a data center in its Loveland, Colorado, headquarters. According to Kai Staats, chief executive officer at Terra Soft, the company began designing the data center earlier this year and finished it off in August. When the final parts for the clusters arrive in November--namely, the PS3 beta machines, which each have a Cell PowerPC processor, two disk drives, and a Gigabit Ethernet port--Terra Soft will plug them all together using Gigabit Ethernet switches. These beta machines, which were used by game makers in lieu of actual PS3 consoles, come in 2U rack-mounted chassis; the alpha PS3 units were actually tower machines that were less useful as a cluster. Sony obviously showed good sense in choosing a rack form factor for the beta PS3 units, knowing they would come back to them as the PS3 launch date approached and that they could--and should--be put to some good use.

Staats will not yet reveal how many PS3 betas will be in the E.coli or Amoeba clusters, but he did say the addition that Terra Soft put onto its data center could house 2,400 1U rack-mounted servers. That means it can hold a maximum of 1,200 2U servers. Staats said only that each cluster would have at least a few hundred nodes, and the exact configuration of the clusters was not yet determined, because they would not even begin building the clusters until November.

While Terra Soft has some preliminary performance information on the Cell chips, he did not want to venture a guess as to how many teraflops the machines might have. "We could estimate, but we want to take the two or three months to run the tests," Staats explained, with a smile you could hear over the phone. With each Cell chip capable of delivering about 200 gigaflops, and each PS3 console having one Cell chip, the aggregate peak number-crunching performance that Terra Soft could put into its data center extension is about 240 teraflops. But, considering that the bandwidth of Gigabit Ethernet links is so low that having more than 500 or so nodes means each node sits around waiting for data, it seems unlikely that Terra Soft will push it that hard. Still, even a 500-node PS3 cluster could deliver 100 teraflops of peak power and maybe 30 to 50 teraflops of performance. (It is very hard to say, since the Cell chip has not been tuned for Linpack or any other benchmark yet.) The amazing thing is that even if two-thirds of the performance of the PS3 cluster went up the chimney, at $599 for a top-end PS3, a 30 teraflops cluster for under $300,000--and one that you can play games on when the super isn't busy--is a remarkable idea.

Terra Soft will use the smaller E.coli cluster--so named because it was the first life form on earth to have its genetic code sequenced by humans--as a test bed for its own software. The Amoeba cluster will be larger (even though, technically speaking, an E.coli bacteria is far more complex, organically, than an amoeba cell) and will be used by researchers at Lawrence Berkeley National Lab and Los Alamos National Laboratory. Other research labs and universities will also be invited to play with the Amoeba cluster, says Staats.

Both clusters will run Yellow Dog Linux V5.0, the company's future Linux release, as well as a beta version of Y-HPC V2.0, which is a cluster-provisioning add-on. Terra Soft has chosen the open source Moab cluster management tool to manage the two clusters.

Y-HPC was originally created for supercomputer clusters based on Power-based Apple Xserve servers, but Terra Soft has expanded its support to Power-based blade JS20 and JS21 servers and System p5 rack and tower servers from IBM. Yellow Dog Linux and the Y-HPC software is also supported on the Power-based and Cell-based blade servers from Mercury Computer. Last June, Mercury partnered with IBM to create a Cell-based blade server, which, according to rumor, IBM is rebadging as the BladeCenter QS20 server.

The researchers at Lawrence Berkeley are being asked to help Terra Soft better tweak its Y-Bio V1.1 gene sequencing software to run on the PS3 cluster. This work will be done early in 2007. Y-Bio is the first application that Terra Soft has created to run atop Yellow Dog Linux, and presumably, there will be more.

IBM, of course, launched the QS20 blade server, which has two Cell chips on each board, several weeks ago in the wake of an announcement that it was building a 1.6 petaflops supercomputer cluster based on Opteron servers and Cell blades for Los Alamos. The U.S. Department of Energy is picking up the tab for this hybrid cluster, but has not provided pricing for the machine. What IBM has said is that the QS20 blades cost $18,995. Two high-end actual PS3 game consoles, which will start shipping November 17, will cost $1,200. This is a huge price differential. The kicker is that IBM is offering an InfiniBand adapter for the QS20, while the PS3 is limited to three Gigabit Ethernet ports. To build a big Cell-based cluster requires a high-speed, low-latency interconnect like InfiniBand. However, Gigabit Ethernet is fine for a smaller cluster--particularly for workloads that do not require lots of messaging traffic between nodes, like gene sequencing. (A cluster of PS3s would not be ideal for modeling the weather or a nuclear reaction.)

Staats says that building a big supercomputer cluster based on PS3s is not what Terra Soft is trying to do, anyway. The appliance-like nature of the PS3 console fits with the rule of thumb in the supercomputer market that states that the simpler the node, the further you can scale a cluster. (The converse of which is the more complex the node, in terms of memory and I/O capacity and bandwidth, the lower you need to push scalability.) So it would seem that the PS3 would be an ideal kind of supercomputer node. But what Terra Soft and its partner, Mercury Computer, believe is that a network of PS3s might be an excellent front end to a giant, high-speed cluster built from Cell-based blade servers that have InfiniBand interconnections. The same software would run on both machines, of course, and would be supplied by Terra Soft.

RELATED STORIES

IBM Delivers Promised Linux-Based Cell Blade Server

IBM to Build 1.6 Petaflops Super for Los Alamos Lab