Sun Fleshes Out Galaxy Opteron Server Line

Published: July 18, 2006

by Timothy Prickett Morgan

It has probably taken a lot longer than Sun Microsystems would have liked, but the company last week finally took the wraps off the next iterations of its "Galaxy" class of Opteron-based servers. With the announcements last week at its Network Computing event in San Francisco, Sun launched a 16-core SMP server, a 10-blade server with an interesting architectural approach, and something that Sun is calling a data server.

Last September, Sun announced its first Galaxy machines with much fanfare, and the X2100, X4100, and X4200 servers, which were designed by returning Sun founder Andy Bechtolsheim, have been instrumental in getting Sun lots of great press and an increasing amount of mindshare among corporate server buyers. Sun's Galaxy designs are less about system architecture perhaps than prior generations of Sun boxes, mainly because they are based on the 64-bit X64 Opteron processors from Advanced Micro Devices. AMD has done so much of the engineering in coming up with the Opteron cores, their integrated memory controller, and the HyperTransport system interconnect for linking the processors to each other and to I/O; nVidia has done the work creating the chipsets that Sun uses in the Galaxies. You might be thinking, so what exactly has Bechtolsheim been doing? Sun's server design team has been focused on the places where it can add value, including the integration of server components (including small form factor disks and low-power power supplies), motherboard design, and the thermal and airflow characteristics of the mechanical design of the server (which is related to the motherboard and which explains why Sun is designing its own boards). By using the Opteron architecture, Sun has drastically reduced the engineering bill for the Galaxy servers, and the value-adds that it is bringing to the machines will help to differentiate the Galaxies from other Opteron-based machines. The Galaxy designs are also intended to allow Sun to do higher volume manufacturing than it is accustomed to doing with its Sparc-based Sun Fire machines, which command a higher premium and therefore can be manufactured in relatively low volumes.

Perhaps the most interesting box that Sun rolled out last week is the Sun Fire X4600. In many ways, its design harkens back to the early days of enterprise-class Sparc servers, which stands to reason given that Bechtolsheim returned to Sun two years ago to create the Galaxy server line.

Back in the days of the UltraSparc-II processors in the mid-1990s, Sun developed a concept for server design that it called the uniboard. Basically, you put one processor and its main memory on a single card and then this plugs into the server backplane. As processor generations change, you leave the backplane the same (as well as its external I/O support) and you simply change the uniboard to incorporate new CPU or memory technologies as they become available. The idea was for a server to span multiple processor generations and for upgrades to be less disruptive to customers. With the UltraSparc-III generation of machines from the early 2000s, Sun moved to server designs that employed either two-socket or four-socket uniboards and their associated main memory. (Back then, all Sparc processors will single core.) The entry and some midrange "Serengeti" servers used the two-socket uniboards (with up to eight in a box) while other midrange boxes and all of the high-end machines used four-socket uniboards.

With the Galaxy X4600 server, Sun is going back to the uniboard concept. Rather than glue off-the-shelf motherboards with two or four sockets together to create the X4600, which scales up to eight sockets, Sun has created a single-socket uniboard of its own design (but still using nVidia chipsets) that can accept the current Rev E generation of Opteron 800 series processors, which come in single- and dual-core versions. Each uniboard has a single processor slot and up to 8 GB of DDR1 main memory (soon to be expandable to 16 GB); three HyperTransport links come off these boards into the X4600 backplane, allowing the processors to talk to local and remote memory in NUMA fashion and to reach out to I/O devices.

The X4600 has uniboards employ three different Opteron processors: the 3 GHz single-core Opteron 856, the dual-core 2.4 GHz Opteron 880, and the dual-core 2.6 GHz Opteron 885. All of these chips have 1 MB of integrated L2 cache per core. According to David Lawler, director of product definition for Sun's Systems Group (which means Lawler is the marketing counterpart to Bechtolsheim's job as head server designer), Sun will have the next-generation "Santa Rosa" Rev F Opterons into uniboards for the X4600, and the system will be able to support boards out beyond that. "We can go quad core with this box," says Lawler. AMD is expected to deliver quad-core Opterons using a 65 nanometer chip process in 2007.

If customers use dual-core Opterons, they can bring 16 cores in the X4600 to bear on a single system image. This will presumably be a very powerful machine, and Lawler says that Sun was readying benchmark tests to prove the X4600's performance as it was preparing the launch. (They were not ready as we went to press, however.) And next year's 32-core X4600 will be a very powerful machine, indeed.

Now, the great question of self-impact, which many Sun shops, Sun competitors, and Sun analysts on Wall Street have been noodling, will be answered. A wise man once said that you have to eat your own lunch before someone else does, and by that logic, then the Galaxy machines should have been to market in around 2002 or so, when IBM and Hewlett-Packard were raiding the Sun base with their Unix and Linux wares with much success. Given the compelling price/performance of larger Galaxy machines, it is reasonable to assume that customers who might have otherwise installed a midrange Sparc-based machine running Solaris 10 will now take a hard look at the X4600. Quad-core Opterons will make this box even more impressive, and will presumably give all but the largest Sun customers a reason to think about Opteron. (Provided their applications can be ported to Solaris 10 on X64, which is not always the case.)

The X4600 has four hot swap disk drives, which are stacked vertically by twos and pushed over to the far right of the server chassis. A vertical CD drive is immediately to the left of these disks, which are hot-swap 2.5-inch SAS drives. The dual 850-watt power supplies sit behind the disks, which keep them from making the area where the CPUs are to the left (and dominating the box) from getting heated by the power supplies and the disks. By moving the disks and power supplies over, the entire front of the server, which is a 4U form factor, can be used to draw air over the processors and main memory to keep it cool. (This is the kind of elegant smarts that Bechtolsheim brings to server design.) The server has an on-board RAID 1 mirroring controller and uses the same Integrated Lights Out Management service processor and N1 Systems Manager software that other Sun Fire servers use. The X4600 has four Gigabit Ethernet ports, six PCI-Express slots, and two PCI-X slots.

Because Sun is operating system agnostic these days, the X4600 supports Sun's own Solaris 10 Unix; Red Hat Enterprise Linux 4, and Novell SUSE Linux Enterprise Server 9; and Microsoft's Windows Server 2003 Standard Edition and Enterprise Edition. VMware's GSX Server and Virtual Server (that's the freebie version of GSX Server) have been certified on the box, too. But the enterprise-class ESX Server hypervisor from VMware is missing from the list at the moment.

According to Lawler, Sun is not selling a configuration of the X4600 with fewer than four uniboards in it. The base box has four dual-core Opteron 880s, 16 GB of main memory, and a single 73 GB SAS drive; it costs $25,995. If you want more clock-speed performance (but fewer threads), you can get an X4600 with four single-core Opteron 856 processors, 16 GB of memory, and two 73 GB drives for $31,995. A heavily configured X4600 consists of eight dual-core Opteron 885s, 32 GB of memory, and two 73 GB disks; it costs $67,495. Sun is taking orders for the X4600 now, and the box will be generally available in mid- to late-July. Knowing this, and the fact that AMD will probably deliver impressive price/performance improvements with the Rev F Opterons in only a few weeks, the current Rev E uniboards could be extremely short-lived--unless, of course, AMD experiences more delays with the Rev Fs than it already has.

The Sun Blade 8000 Modular System is Sun's second stab at the blade server market--its first foray being utterly forgettable and short-lived. This time around, Sun might get better traction. With the current blade designs from IBM and HP, which together have the lion's share of blade server shipments with their respective BladeCenter and BladeSystem machines, blades plug into a backplane that allows for management of blades at the chassis level. To talk to the outside world, each blade has mezzanine cards that link to external SAN and NAS arrays for storage and Ethernet or InfiniBand switched fabrics for linking to each other or to outside devices. Bechtolsheim took a slightly different approach with the Sun Blade 8000, and the network backplane virtualizes the PCI-Express peripheral links on each server blade. No mezzanine cards, no backplane for systems management. The blades in the Sun Blade 8000 have the same ILOM service processor and the same N1 Systems Manager software that all other Sun Fire servers use.

By doing it this way, each blade is a server that has virtualized I/O that is handled by the chassis. If you yank a blade out and plug a new one in, all of the I/O associated with the old blade will activate on the new blade. Basically, the design extends and aggregates the PCI-Express slots on a server into the chassis. PCI-Express peripheral cards that link to SAN and NAS arrays, for instance, plug into cages in the back of the chassis, just as they would in a normal server.

The Sun Blade 8000 is not a small machine, given its 19U form factor, but it does support up to 10 four-socket X8400 blade servers and up to six fully redundant power supplies. The X8400 blades only support dual-core Opterons, including 2 GHz 870s, the 2.2 GHz 875s, and the 2.6 GHz 885s. These blades have two hot-swap 2.5-inch SAS or SATA drives to store OS images (if you want to do it locally) and a RAID mirroring controller onboard; they also support up to 16 GB of DDR1 main memory per CPU socket, for a total of 64 GB per blade. The same Solaris, Linux, and Windows versions that were supported on the X4600 are supported on the Sun Blade 8000 and its X8400 blades. The Sun Blade 8000 chassis costs $4,995, while an X8400 blade server with four dual-core Opterons and 8 GB of main memory costs $14,600. The Sun blade server will be available in mid-July.

By the way, it seems likely that at some point, Sun will deliver an X8400 motherboard tipped on its side and create a rack-mounted server, probably with a 2U or 3U form factor, called the X4400. It also seems reasonable that Sun might deliver a machine with two-socket Opteron blade servers, and could even create UltraSparc-IV versions of the blade machine.

The last new Galaxy server announced last week was something that Sun is billing as a "data server," and some people will call it a skinny server with a lot of integrated disk or a reasonably capacious yet compact storage array that has Opteron brains and therefore can be used to do specific kinds of processing work along with managing large amounts of storage. The X4500 Data Server, which was developed under the code-name "Thumper," and it is basically a revamped version of the X4100 two-socket Galaxy server that has been rejiggered to have a whopping six eight-port SATA disk controllers on the board. By having so many SATA ports, the Thumper array can house up to 48 3.5-inch SATA disk drives as well as two dual-core Opteron 285 processors, which run at 2.6 GHz, into a 4U form factor. The Thumper server supports 250 GB or 500 GB SATA drives, so it can support a maximum of 24 TB of data.

Lawler says that by using onboard SATA controllers, this server can deliver 2 GB/sec of sustained I/O from disk to memory, which would take as many as eight 2 GB Fibre Channel adapters in an external SAN array. Those FC adapters are not cheap, so getting rid of them can make an array a lot less expensive for a particular server. Moreover, he claims that the Thumper server offers anywhere from two to five times the density of comparable server/array configurations, which means you can rack 'em and stack 'em. In fact, the 100 teraflops Opteron-based supercomputer that Sun and NEC are building for the Tokyo Institute of Technology has 42 of these Thumper arrays in addition to a whole bunch of the X4600 servers outlined above. Because the X4500 has processing capacity as well as massive storage capacity, Lawler expects that other high performance computing customers will be interested in the units, particularly if they want to run data analytics on slices of data sets. He also expects the machines to be useful for surveillance applications and anywhere that customers want to run analytics on large data sets. The machine might prove very interesting for clustered file systems. But it is not a generic server, in as much as it only supports Solaris 10 and its Zettabyte File System. RAID-Z data protection is implemented in software on the device, which also cuts out the cost of RAID hardware controllers.

The X4500 Data Server will be available in mid-August. An X4500 with 48 250 GB disks (for a total of 12 TB of space), two 2.6 GHz dual-core Opteron 285s, 16 GB of main memory, and Solaris 10 and ZFS locked and loaded costs a relatively modest $32,995. Switching to 500 GB disks jacks the price up to $69,995. Lawler says that Sun is giving discounts to customers who buy 10 or more of the Thumpers and at that price, it gets the acquisition cost down below $2 per GB.

One last thing. Any expected upgrades to the UltraSparc-IV+ processors, which are currently running at 1.5 GHz, did not make it to market as part of this Network Computing extravaganza. Sun was expected to boost the clock speed of these chips sometime in mid-2006.