New Power6+ Iron: The Feeds and Speeds

Published: May 4, 2009

by Timothy Prickett Morgan

In last week's issue of The Four Hundred, I gave you the scoop on the Power6+ rack, tower, and blade server announcements that Big Blue was making at the COMMON midrange trade show out in Reno, Nevada. A lot of the details about the new machines were not available at the time, and I have done some digging to get you more information on the new boxes.

(That story talked about a lot more than the new boxes, so if you missed it, you should check it out here.)

You will recall that IBM announced that it was plunking its Power6+ processors, running at slightly higher clock speeds than the Power6 chips they replaced, in the entry Power 520, the midrange Power 550, as well as a new blade server called the JS23/JS43. As far as I know, no one else in the IT trade press has figured out that these machines, as well as some of the machines launched back in October 2008, are based on Power6+, not Power6, chips. (In a separate story in this issue I explain the differences between Power6 and Power6+ chips and go into the prospects of a speed bump on Power6+ machines later this year in advance of Power7 chip due next year in commercial machines.) IBM certainly didn't help make the whole Power6+ issue clear, and has scrubbed all of the pluses out of its press releases and most of its documentation, as I expected.

No matter. This is why you keep me around after all these years. (It will be 20 years in the first week of July, if you can believe it.)

The Power 520 with Power6+ Chips

The new Power 520 server comes in the same 4U chassis that the 520 boxes have been using since the converged i-p Power6 line was launched back in April 2008. As it turns out, the faster Power6+ chip, which runs at 4.7 GHz, is only available in machines with two or four cores activated. The earlier Power6 configurations offered a single processor card machine with one or two 4.2 GHz Power6 cores activated and then a two processor card box with either two or four cores activated. IBM is not offering a single-card version with the Power6+, for whatever reason. The processor packaging for the Power6+ chip used ion the Power 520 box also has a single 32 MB L3 cache module on the chip package, an L3 cache that was used on larger Power6 boxes in the past. (The Power 595 has two L3 caches for each dual-core processor, while Power 550s, 560s, and 570s as well as the new JS23/43 blade has one L3 cache chip shared by both cores.) This L3 cache is one reason why the new 520 has more oomph than the old 520.

But on the Commercial Performance Workload (CPW) test used to calculate relative performance for OS/400, i5/OS, and i workloads, the cache doesn't seem to make as much difference as on the rPerf test used for AIX workloads. On a two-core 520 running i 6.1, the move from the earlier to the newer box resulting in 14.5 percent more CPWs--to 9,500 CPWs to be specific--while on the rPerf test running on AIX boxes, the improvement in performance is 26.2 percent--to 20.13 rPerfs. The top-end four-core Power6+ machine is rated at 18,300 CPWs, about 17.3 percent more oomph than the Power 520 using 4.2 GHz Power6 chips.

The new Power 520 using the Power6+ chip comes with the same six 3.5-inch disk bays (either SAS or SATA drives can be used) and DVD bay as earlier machines. IBM is also offering an optional disk bay that holds up to eight 2.5-inch small form factor SAS drives (spinning at 10K or 15K RPM) for customers who want to lower the heat in their systems since these smaller disks generate about half the heat as their larger 3.5-inch predecessors. IBM is also allowing 3.5-inch or 2.5-inch solid state disks (which I detail within this issue here) to be plugged into the new as well as existing Power 520s. The system board has an integrated SAS and SATA disk controller and an optional mezzanine card that gives the controller RAID 5 data protection. The Power 520 supports from 4 GB to 32 GB of main memory on the two-core version (which is a single Power6+ chip on a processor card with four DDR2 main memory slots) and from 4 GB to 64 GB on the four-core version (that's two of the same processor cards). The Power 520 has three PCI-Express and two PCI-X peripheral slots, an integrated dual-port, Gigabit Ethernet Virtual Ethernet NIC; a quad-port Gigabit Ethernet NIC or dual-port 10 Gigabit Ethernet NIC are optional.

The Power 520 system has two GX bus slots, which are used to connect remote I/O drawers to the server. The single-core machine doesn't offer external I/O, but the two-core machine has one I/O loop and the four-core machine has two loops. Each loop can support a different number of I/O drawers depending on the kind of peripherals used and the type of link used to make the loop - Remote I/O (RIO), InfiniBand, and InfiniBand 2 are the technologies the Power Systems line currently supports. (IBM calls InfiniBand ports 12X for some reason in some of its documentation, just like it called its modified Fibre Channel links RIO to try to keep people guessing.) Anyway, on the Power 520, RIO supports six drawers per loop and 12X supports four drawers per loop using PCI-X peripherals in the drawers and two drawers in the loop using PCI-Express peripherals in the drawers.

According to the IBM documents I have seen, using a Power6+ setup and 667 MHz DDR2 main memory and 12X double data rate peripheral links, the Power 520 has 75.2 GB/sec of bandwidth coming out of its L1 caches and 188 GB/sec coming out of its L2 caches. On the L3 cache linked to each dual-core Power6+ chip, the bandwidth is 37.6 GB/sec, and there is 16 GB/sec of bandwidth between those four memory slots on the processor card into the on-chip memory controller. IBM says that the total I/O bandwidth on the Power 520 is 28.2 GB/sec, which is 4.7 GB/sec on the internal I/O bus, 4.7 GB/sec on the first GX slot, and 18.8 GB/sec on the second GX slot. All that bandwidth is necessary to feed the possibly large number of disk drives that can be attached to a Power 520--in the case of i operating systems, using 428 GB disks you can put 125 TB of capacity--that's 296 drives--on all of those remote I/O drawers.

The new Power6+ Power 520 costs $14,950 for the base iron for the AIX edition; that machine has two 4.7 GHz cores activated and 4 GB of main memory and two 146.8 GB disks. AIX costs $150 per core and Software Maintenance costs $299 per core, for a total of $15,848. Adding a second processor card with the cores activated and pushing memory up to 8 GB, plus the AIX and Software Maintenance on the cores, raises the price up to $22,906. Why am I telling you AIX Edition prices? Because IBM isn't publicizing i prices on the Power6+ version of the Power 520, I have to do some guesswork. If i 6.1 costs the same, at $995 per core for the base OS and database license plus $1,250 for the first five users and $750 per processor for Software Maintenance, then the new Power 520 with two 4.7 GHz cores, 4 GB of memory, and two disks should cost $19,690. (Remember, the AIX machines do not have database software.) With four cores and 8 GB of memory running i and the same five users should cost $29,190 for five users.

The Updated Power 550

The Power 550 chassis is a similar 4U box with the same options in terms of internal I/O as well as the number of RIO and InfiniBand loops and ports for external I/O. But instead of having only two processor cards, the Power 550 has up to four, for a total of between two and eight cores. (This is true whether you are using Power6 or Power6+ chips). Each processor card has eight DDR2 memory slots, for a total of 64 GB of main memory per card and 256 GB of main memory per system. On the 3.5 GHz Power6 chip cards running at 3.5 GHz, IBM tops the memory out at 128 GB, but the 4.2 GHz Power6 and the new 5 GHz Power6+ cards have the full 256 GB of main memory. A single 32 MB of L3 cache is on all cores. The other big difference between the Power 520 and Power 550 is that the latter machine is allowed to have up to 584 disk drives attached to it through its RIO and 12X links. Using 428 GB disks formatted for i 6.1, that works out to 249 TB of capacity--something no customer is actually doing, I am pretty sure.

With the Power6+ chip revving at 5 GHz, the Power 550 offers from 10,600 CPWs of performance on a machine with two cores up to 37,950 CPWs on a machine with eight cores. That's between 14.2 percent and 20.8 percent more bang for the buck, depending on the cores.

Because of the slightly higher clocks speeds of the Power6+ chip used in the Power 550, the memory bandwidth numbers are higher. There's 80 GB/sec coming out of L1 caches into their cores, and 200 GB/sec coming out of each L2 cache. L3 cache has 40 GB/sec of bandwidth, and the intra-node buses that link the processor cards together run at 20 GB/sec. Now, the internal I/O bus bandwidth is also slightly higher, at 5 GB/sec, as is the first GX slot (which is a passthru bus), at 5 GB/sec, but the second GX slot is running slower, at 13.3 GB/sec, than on the Power 520, even though the Power 550 can, in theory have twice as many disk drives. Beats me why this is the case. I just report the news.

As with the case with the new Power6+ version of the Power 520, IBM is not giving out prices (at least not through its online store, and only some features of the machines have prices in the announcement letters) for the updated Power 550. But AIX shops get pricing because--well, I don't know why. There's no good reason for any vendor to not provide list prices for all of their products. Ever.

Anyway, a Power 550 with two 5 GHz Power6+ cores, 4 GB of memory, and two disks costs $35,953. AIX costs $385 per core and a year of Software Maintenance costs $449 per core. The heaviest base configuration of the Power 550 using the new chips, with eight cores, 16 GB of memory, and two disks, costs $115,205. As far as I know, the Power 550 running the i 6.1 operating system does not have user-based pricing for the operating system and database combination, and IBM is still charging $44,000 per core for the software (including Software Maintenance, which costs $4,000 per year) with an unlimited number of end users. So that base two-core box will set you back $123,953, and that eight-core box would be $467,205. That software does not include 5250 enablement for green-screen legacy applications, which costs $50,000 per core (or $150,000 to turn on all cores in the box). So the max configuration of the Power 550 would be $617,205 with all the cores set up to run i 6.1 and do green-screen software.

The Power 560--It Was Also A Power6+ Machine

Just an aside: What makes a Power 560, which has up to 16 cores, different from a Power 550, which has up to eight cores, is that IBM gears down the Power6+ chips to 3.6 GHz and puts two whole chips (that's four cores) on each processor card; this card also has a dozen instead of eight DDR2 memory slots, which boosts the memory per card to 96 GB, for a total of 384 GB per system. The thing about the Power 560 is that because it has twice as many cores as the Power 550, it can support twice as many logical partitions, in this case, up to 160. To be balanced, the processor cards probably need 16 memory slots, but again, I am no system engineer. The machine offers one more RIO or 12X loop and up to 1,332 disk drives, for a maximum of 570 TB of capacity for i 6.1. The CPWs on the Power 560 range from 14,100 on a four-core machine to 48,500 on a 16-core box. With IBM charging for i 6.1 by the core--and charging quite a bit--this machine is really for customers wanting to consolidate lots of AIX or Linux workloads on a box with modest i workload requirements.

The New JS23/JS43 Blades

The new four-core JS23 blade server fixes a number of issues with the earlier JS22 blade, namely memory capacity, I/O bandwidth, and symmetric multiprocessing scalability. Last year, when IBM finally converged the i and p product lines, the single-socket JS12 and two-socket JS22 blades were better than no blades at all, and were priced fairly against Power 520 and Power 550 machines running i5/OS V5R4 and I 6.1 that had their core counts cut in half compared to the AIX versions. (Yeah, I know. You could just scream sometimes, right? In October last year, IBM finally made i 6.1 available on the same exact boxes as the AIX version, which was a step in the right direction.)

The new JS23 blade is a two-socket blade, like the JS23, but it offers twice as much main memory--from 4 GB to 64 GB--which is important for virtualization. Because of the faster clock speed on the Power6+ processor running at 4.2 GHz on the JS23, memory to chip bandwidth is higher, at 21.3 GB/sec per chip, and all the L1 and L2 caches are proportionally offering more bandwidth. Also, the JS23's Power6+ chip has that 32 MB of L3 cache, which was missing from both the JS12 and JS22 blades and therefore hurt performance on workloads (particularly Java and PHP, which are interpreted rather than compiled languages). The JS12 and JS22 also had a PCI-X peripheral bus that topped out at 5.8 GB/sec of bandwidth, which was a lot less than the equivalent 14 GB/sec on the original Power 520 and 550 rack and tower machines from last year. The JS23 uses PCI-Express buses (with two PCI-Express daughter cards) to link out to peripherals, and these buses packs 8.4 GB/sec of I/O bandwidth. The JS23 has a dual-port Gigabit Ethernet port and a SAS disk controller to link to its single 2.5-inch SAS or SSD drive.

If you need more I/O bandwidth for i-based workloads running on blades, you can snap two JS23 blades together like mobile homes and, using an XMP symmetric multiprocessing port, create a "double-wide" blade server that IBM calls the JS43 when you do that. The JS43 doubles up Power6+ cores, main memory, Ethernet ports, disk, and so on.

The JS23 is rated at 14,400 CPWs with all four cores activated, which is only 4.3 percent more than the JS22 it replaces, and considering that the JS22 didn't have L3 cache, I am surprised that the difference is not larger, as it was on the Power 520 detailed above. The JS43 with all eight cores running is rated at 24,050 CPWs, which is about where a three-core Power 550 would be if you could get one. (You can't.)

The base configuration of the AIX Edition of the JS23 blade comes with two 4.2 GHz Power6+ cores, 4 GB of main memory, and a single 73.4 GB 10K RPM 2.5-inch SAS drive. It costs $8,919, and AIX costs a mere $85 per core with Software Maintenance for one year costing $299 per core. A BladeCenter S chassis (the one that runs on 120-volt power, has six blade slots, and is aimed at SMBs and their office environments) costs $4,499 while the BladeCenter H chassis (240-volt power with 14 blade slots) costs $3,849. The JS23 i Edition is the same configuration, but adds the required Fibre Channel and SAS expansion cards to connect to storage (i 6.1 needs this to link to storage, even as the Virtual I/O Server is what the operating system sees as the storage for the blade); that puts the base price for a JS23 blade running i at $9,917. It costs $14,995 to put i 6.1 on one of the processor cores on the JS23 blade, plus $2,500 for 10 end users. So on a four-core JS23 blade setup for i applications, we're talking $27,412 for one core aimed at i apps, and $72,397 for i on all blades. If you wan to make a JS43, you slap two JS23s together. I am not certain what 5250 enablement costs, but it should be the same as on the Power 550 above.

These new Power6+ machines will be available on May 22, and they do not support i5/OS V5R4 and only run i 6.1. The new machines do support the earlier AIX 5.3 release as well as the current AIX 6.1, and also support Novell SUSE Linux Enterprise Server 10 SP1 or later and Red Hat Enterprise Linux 4 Update 5 or Enterprise Linux 5 Update 1.

RELATED STORIES

IBM Launches Power6+ Servers--Again

Power Systems Down A Bit in IBM's First Quarter

Come On Out, Power6+, You Win

Power vs. Nehalem: Time to Double Up and Double Down

Power vs. Nehalem: Scalability Is So 1995, Cash is So 2009

IBM and Resellers Do the iLoyalty Blitz

i Roadmaps: Here Be Dragons

IBM Doubles the Cores on Midrange Power Systems

Intel Keeps Both Arms Swinging with Xeons, Jabs with Itanium

More Power7 Details Emerge, Thanks to Blue Waters Super

Intel's Nehalems to Star at IDF, AMD Pitches Shanghai

Intel Talks Up X64, Itanium Roadmaps Ahead of IDF

Intel Announces First "Penryn" Xeon Processors

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot