IBM Readies Super-Dense 16-Way p5 Rack Server

by Timothy Prickett Morgan

IBM said this week it is working on a new variant of its p5 575 server for high performance computing workloads that will cram an incredible 16 Power5 processor cores (that's eight physical Power5 chips) in a 2U form factor rack-mounted server. The advent of such a machine must be something of a record of computing density, and while records are interesting, Big Blue is clearly building such a server because someone wants to pay for it.

IBM has been shipping an eight-core p5 575 since mid-February, which uses Power5 chips running at 1.9 GHz. This p5 575 has 4 GB DIMMs that allow it to support up to 256 GB of main memory, plus 36 MB of L3 cache memory per dual-core Power5 chip. The initial p5 575 server had four Gigabit Ethernet ports for connectivity and has just been given support for IBM's "Federation" High Performance Switch (HPS); it requires a secondary I/O drawer to support PCI-X peripherals. This drawer can support up to 20 PCI-X cards. IBM plans to. IBM is supporting AIX 5L 5.2 and 5.3 on the p5 575 server, as well as Novell's SuSE Linux Enterprise Server 9 and Red Hat's Enterprise Linux 3 AS. All of these operating systems, thanks to IBM's own Virtualization Engine hypervisor, can be run concurrently within logical partitions on the p5 575 (as is the case with all p5 servers).

In February, IBM was shipping the p5 575 with eight cores running at 1.9 GHz in 16-node clusters with a total of 128 processor cores, and at the end of April the clustering support was extended to 128-node clusters for a total of 1,024 cores. While the p5 575 comes in a 2U form factor, which should mean that up to 24 servers should be able to be housed in a single rack, heating issues and the need to attach peripherals restrict the density to 12 nodes per rack. That works out to just under 700 gigaflops per rack. If IBM could get only p5 575s in the rack, stacked with no space, it could cram almost 1.4 teraflops in a rack using the 1.9 GHz processors. But, it can't.

So creating the 16-core p5 575 using slightly slower Power5 processors--which burn a lot less electricity and throw off a lot less heat per core--is just about the only way that IBM can increase the density (in terms of flops per rack) of the p5 575. To that end, IBM has reworked the p5 575 motherboard and added another four processors to it while still maintaining the same basic peripherals (main memory, disk bays, and PCI slots) as the original p5 575. To keep the 16-core machine from melting, IBM is still only putting 12 servers per rack and is also stepping down the clock speed of the Power5 chips in the 16-core box from 1.9 GHz--the fastest Power5 core--to 1.5 GHz--the slowest Power5 core. (Power5 chips are also available in 1.65 GHz versions as well.) By doing so, IBM can boost the number-crunching power of a rack of p5 575s to about 1.1 teraflops (by my math, not IBM's)and can double the number of cores in a 128-node cluster to 2,048. With the full complement of cores, a cluster of the new 16-core p5 575s would deliver about 11.75 teraflops of aggregate peak computing power.

IBM said it will deliver the 16-core p5 575 in the fourth quarter, and added that for customers who have compute-intensive workloads, this machine will be a better choice than the current eight-core box. For those customers whose workloads require more memory bandwidth per processor, the earlier eight-core box will be a better choice, since it can allocate twice as much main memory and memory bandwidth per per Power5 core as the new p5 575 will be able to deliver.

As I previously reported, IBM just sold a 5 teraflops cluster of the p5 575 servers to the Max Planck Society, a consortium of German research institutions, which will double the computing power that the institutions have at their disposal to run simulations to around 10.2 teraflops. The bulk of the 86 eight-way machines will be installed at the Fritz Haber institute, which is doing research in materials science, including various nano-materials. The Max Planck Society already has a cluster of 32 big pSeries SMP servers that delivers just over 5 teraflops of computing power. Lawrence Livermore Laboratory will eventually have largest p5 575 cluster in the world when it upgrades the ASCI Purple machine IBM built several years ago for the U.S. Department of Energy to the Power5 processors. When it goes operational in the second quarter of 2006, it will have 1,536 of the eight-core P5 575s, for a total of 12,288 cores, and will be rated at 93.4 teraflops. Lawrence Livermore has three other Power-based systems in the ASCI Purple cluster that will boost its aggregate processing capacity to over 100 teraflops.

With the eight-core motherboards, the base p5 575 costs $4,575; the four Power5 processors cost $40,000 (that's $5,000 per core), the system rack costs $10,000, and the I/O drawer costs $32,000. A 4 GB DIMM costs $8,448, while a 16 GB DIMM costs $61,579. IBM did not provide pricing information on the forthcoming 16-core p5 575.