SGI Launches Blade-Style Altix Linux Supers

Published: June 26, 2007

by Timothy Prickett Morgan

The International Supercomputing Conference 2007 event, which is the biggest supercomputing show in Europe and one of two big events in the world, is taking place this week in Dresden, Germany. There will be a lot of announcements, including the semi-annual Top 500 ranking of supers. But one of the bigger announcements this week will be coming from Silicon Graphics, which is launching a new blade-style super, code-named "Carlsbad," based on a variant of the "Atoka" motherboard.

The Atoka board was previewed last fall by Intel, which partnered with SGI and server motherboard maker Super Micro to design a motherboard that was skinny enough that you could cram two of them into a single server chassis. Because server cases are a long-since standardized form factor (blame the military from the 1940s for the 19-inch wide rack), it seems unlikely that server makers or customers want to change that, which is why with the Atoka family of boards, Intel and its partners are trying to cram two servers where one used to be. You get double the density in the end, which is what matters to most customers--particularly the largest supercomputer centers in the world.

The new Altix Integrated Compute Environment is another step in the evolution of SGI, which has created a respected Altix platform using NUMA clustering of Itanium processors to create very large and very powerful global memory clusters that run Linux. But not everyone wants Itanium processors, and not everyone wants to run Linux these days. Some companies want much cheaper clusters than an Altix 4700 Itanium box, and some want to be able to run Windows as well. And to that end, SGI last year rolled out rack-based X64 servers. Those products were merely placeholders to get SGI into the X64 game; the Altix ICE boxes have engineering in them that sets them apart from normal rack boxes.

Back in March, SGI quietly announced the Altix XE310 servers, which used a variant of the Atoka board called Atoka-V. This is a normal server motherboard, in this case based on the Intel 5000P chipset and delivering two processor sockets that support either the dual-core "Woodcrest" Xeon 5100 and quad-core "Clovertown" Xeon 5300 processors; the board had integrated InfiniBand and Gigabit Ethernet ports on the board, which is why supercomputer customers are excited about Atoka motherboards. With InfiniBand on the board, you don't have to pay extra for a host adapter, and supercomputer customers are cheapskates when it comes to hardware. Which stands to reason when you are buying thousands of servers.

The Carlsbad boxes are a totally different twist on the Atoka idea, merging the dense packing these boards allow with a blade concept. The Atoka-P board is not meant to be a standalone board, like the Atoka-V, but rather has a backplane connector that is meant to plug the machine into a blade server chassis. According to Bill Mannel, senior director of marketing at SGI, Intel and SGI were chasing a deal at the U.S. Department of Energy that they lost to IBM, which won the deal with its hybrid BladeCenter-Cell supercomputer, nicknamed Roadrunner since it is being built at the Los Alamos National Lab in New Mexico. (See IBM to Build 1.6 Petaflops Super for Los Alamos Lab for more on that deal.) In the process of pitching for the Los Alamos deal, Intel and SGI pitched some of the ideas that ended up in Carlsbad, and in the interest of time to market, the two enlisted Super Micro to make the boards.

The Carlsbad Altix ICE chassis is a 10U rack-mounted form factor box, which SGI calls an individual rack unit. Each blade is based on a single Atoka-P board, and a total of 16 of these boards are mounted in two horizontal stacks in the center of the chassis. Two chassis management controllers sit in the chassis below these blades. Two rows of front-loaded power supplies (eight in total) run along the outside edge of the chassis, and sitting between these power supplies and the Atoka blades is a series of 4x speed InfiniBand switch blades. The whole shebang can support up to 128 processor cores using the Clovertown chips and plugs together with not a single cable between the components. The InfiniBand interconnect creates a 3D torus topology, which is how many other clusters are arranged these days.

The Atoka-P blade is based on the 5000X chipset from Intel (code-named "Greencreek"), which supports two processor sockets and two 4x InfiniBand ports on embedded host channel adapters. The dual InfiniBand channels are for bandwidth and for redundancy. Customers can--and often do--use one set of InfiniBand links for systems management and one to lash the machines together into a cluster. Each blade has eight memory slots, which means it maxes out at 32 GB using 4 GB DIMMs but, given the stinginess of supercomputer customers, will have a practical maximum of 16 GB per blade using the much less expensive 2 GB DIMMs. (Of course, in a year, 4 GB DIMMs will probably be a lot less costly.) Because SGI was interested in absolute density, the Carlsbad blades do not have local storage. Storage links into each blade through InfiniBand.

Four of the Altix ICE boxes can be put into a single rack, according to Mannel, and using the fastest 2.66 GHz Clovertown Xeon processors, that rack can deliver 512 cores all linked with InfiniBand and about 5.32 teraflops of computing power. Such a rack would burn about 39.5 kilowatts of juice and weigh just under a ton. (250 pounds per square foot is the upper limit in most data centers, and the Carlsbad rack weighs in at 246 pounds per square foot.) The Carlsbad machines use a lot of the efficient power supplies and cooling architecture of the Altix 4700 Itanium boxes, and can be equipped with water-cooled doors that link into standard data center water chillers.

A full rack of the Altix ICE machines costs around $350,000, depending on processor and memory options, and Mannel says that the practical lower limit to get into the architecture is a half rack of boxes. The design goal for the Carlsbad blade design was to keep the price premium compared to standard rack servers down to 10 percent or lower.

Novell's SUSE Linux Enterprise Server 10 is certified on the Carlsbad supers today, and Red Hat Enterprise Linux 5 should be certified on the box in about six months. SGI's ProPack for Linux math libraries will also work on the Carlsbad machines, and customers can use SGI's homegrown cluster tools or those from Scali to manage their clusters. Intel's Fortran and C++ compilers plus its VTune and Math Kernel libraries are also supported, as are all of the popular variants of the MPI protocol used to let nodes in clusters talk to each other. And Altair's PBS Professional 9.0 software is available to manage workloads running on the cluster. And, given SGI's recent partnership with Microsoft, it would not be surprising to see Windows Compute Cluster Server certified to run on the Carlsbad machines soon.

The Altix ICE servers will be available on July 27. General Atomics, which is based in San Diego, and the University of Exeter, which is based in the United Kingdom, are two customers who have had early access to the Altix ICE boxes.

RELATED STORIES

NASA to Replace 'Columbia' Itanium-InfiniBand Cluster

SGI and Microsoft Partner on Windows Supercomputer Clusters

Intel Crams Two Half-Size Motherboards into a Chassis

SGI Comes Out of Bankruptcy, Says It is Lean and Mean

SGI Launches Entry Woodcrest Xeon, Itanium Linux Supers

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot