NASA Buys Big Xeon-Linux Cluster from SGI

Corrected: September 25, 2007

by Timothy Prickett Morgan

NASA's Ames Research Center is shelling out more money as it upgrades its computational capabilities, and this time around the center's long-time collaborator on supercomputers, Silicon Graphics, has won the deal.

SGI and NASA Ames, which is located in Moffett Field, California, and which provides computational power for the space agency's research and design projects, announced this week that Ames will be installing a 4,096 core Linux cluster based on Intel's quad-core "Clovertown" processors as embodied in SGI's new blade-style Altix Integrated Compute Environment, or Altix ICE, designs.

The ICE boxes--you wait a long time for a joke like that--were announced by SGI back in June, under the code-name "Carlsbad." They are based on a motherboard co-designed by Intel, Super Micro, and SGI called the Atoka-P, which occupies half the width of a standard server chassis and has integrated InfiniBand networking on the board. The Atoka-P boards have room for two Clovertown processors, and are based on Intel's "Greencreek" 5000X chipset. This board implements two 4x InfiniBand ports on embedded host channel adapters. Each blade has eight memory slots, which means it maxes out at 32 GB using 4 GB DIMMs. The Carlsbad blades do not have local storage, but rely on InfiniBand for links out to storage. An Altix ICE chassis is a 10U rack-mounted form factor box, which SGI calls an individual rack unit. Each blade is based on a single Atoka-P board, and a total of 16 of these boards are mounted in two horizontal stacks in the center of the chassis. A single management controller sits in the chassis below these blades. Two rows of front-loaded power supplies (eight in total) run along the outside edge of the chassis, and sitting between these power supplies and the Atoka blades is a series of 4x speed InfiniBand switch blades. The whole shebang can support up to 128 processor cores using the Clovertown chips and plugs together with not a single cable between the components. The InfiniBand interconnect creates a hypercube topology.

According to Bill Thigpen, who is engineering branch chief for NASA Advanced Supercomputing and the guy who puts systems inside the Moffett Field data center, NASA is exploring its options when it comes to running code these days. That's why NASA Ames announced it was adding a 6 teraflops Power5+ System p cluster from IBM at the end of June. "One of the things that we have realized is that there is not a perfect system for all code," explains Thigpen. "So we are now going through the evaluation process to see which NASA codes run best on what architectures. Based on what we learn, we will grow our systems. We are all about doing science, and we will pick the machines that help us do science."

So far, IBM has not pitched Linux-based Blue Gene/L massively parallel supercomputers to NASA Ames, but Thigpen conceded that if he thought any of NASA's code would run best on that kind of machine, he would have no qualms with choosing it over any other style of supercomputer.

The largest machine at NASA Ames is the "Columbia" Itanium-InfiniBand cluster, which runs Novell's SUSE Linux Enterprise Server. Columbia is actually three different generations of Altix servers, with a total of 10,240 cores. The first piece was comprised of air-cooled Altix 3700s, followed by Altix 3700 BX2 machines, which came equipped with water jackets to take the heat out of the racks. The Altix 4700 portions of the Columbia cluster also have water jackets.

Later this month, NASA Ames will install a new Itanium-based Altix cluster with 2,048 cores, which will use the Altix design's NUMA-style shared global memory and which will represent the largest single instance of Linux running in the world. This machine, which has not been named yet since it is not on the floor, will not have water jackets. This new cluster uses Intel's dual-core "Montecito" Itanium 9000 processors, but could perhaps soon to be upgraded to the not-yet-shipping "Montvale" kickers to the Montecitos. (Such an upgrade might be in store for Columbia as well.)

The 4,096-core Altix ICE machine, which is not part of Columbia and which does not have a name yet either, will have water jackets from the get-go and super-efficient power supplies, too. It will need these features because of the intense power density of the Altix ICE blade design, which doesn't just include those 1,024 Clovertown chips, but also 4 TB of main memory. The Altix ICE cluster being installed by NASA Ames is rated at 43 teraflops and occupies a mere eight server cabinets--each burning about 30 kilowatts of juice; it should be installed by the end of the month or maybe in October, depending on how fast the plumbers work.

"It just blows me away that all of that will fit into eight cabinets," says Thigpen.

The data center at NASA Ames is about the size of three basketball courts, and currently has a maximum power draw, including power and cooling, of 2 megawatts. Over the next 12 to 14 months, the facility will be upgraded to support 4 megawatts of power, and over the long haul will be expanded further to 6 megawatts. NASA has a lot of number crunching to do to get men back to the Moon and on to Mars as well as to do basic aeronautical research. Interestingly, one of the things that the new Altix ICE cluster will be doing is running simulations to help NASA understand how aircraft might fly in the thin atmosphere of Mars.

This story has been updated since it was originally published. The Atoka-P board was designed by Intel, Super Micro, and SGI, not just by Intel and Super Micro. The ICE Box has a single chassis management controller, not two. And the cluster that NASA is building is a hypercube topology, not a 3D torus. IT Jungle regrets the errors. [Corrected 9/25/07]

RELATED STORIES

NASA to Replace 'Columbia' Itanium-InfiniBand Cluster

SGI Launches Blade-Style Altix Linux Supers

SGI and Microsoft Partner on Windows Supercomputer Clusters

SGI Sues Graphics Chip Maker ATI Technologies

SGI Comes Out of Bankruptcy, Says It is Lean and Mean

SGI Launches Entry Woodcrest Xeon, Itanium Linux Supers

NASA to Acquire Huge SGI Altix Linux Cluster

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot