PCI-Express 4.0 Spec To Double Up Peripheral Bandwidth

December 5, 2011 Timothy Prickett Morgan

Systems using PCI-Express 3.0 peripherals are not yet on the market–unless you count Xeon E5-based machines shipping ahead of next year’s launch–and the basic feeds and speeds of the PCI-Express 4.0 spec have been hammered out by the PCI-Special Interest Group that controls the spec.

After about nine months of running simulations, PCI-SIG said that it would be able to push the bandwidth on the PCI-Express bus to 16 GT/sec (that’s gigatransfers per second), double the 8 GT/sec of the shiny new PCI-Express 3.0, and still remain on copper interconnects between adapter cards and peripheral slots on systems. This will be the last hoorah for copper, though. After this, we will need to move to optical interconnects, and that will present a whole new set of electrical engineering challenges.

The original PCI-Express 1.0 design had a 2.5 GT/sec transfer rate and delivered 250 MB/sec of throughput per lane using 8b/10b encoding. So an x16 slot topped out at 4 GB/sec of bandwidth; most disk controllers and network adapters crafted in PCI-Express Gen 1 used an x8 slot and therefore topped out at a mere 2 GB/sec. Current PCI-Express 2.0 slots run at 5 GT/sec, use the same 8b/10b encoding, and can push data at 500 MB/sec per lane. By this time, some peripherals, such as video cards and GPU co-processors, required the x16 Gen 2 slot, which topped out at 8 GB/sec of aggregate bandwidth in Gen 2 slots. You can also see why the advent of InfiniBand networks running at Quad Data Rate (QDR) speeds of 40 Gb/sec and now Fourteen Data Rate (FDR) speeds of 56 Gb/sec have necessitated an upgrade to PCI-Express 3.0. That’s 40 gigabits per second, and the PCI-Express Gen 2 x8 slot topped out at 32 gigabits of raw bandwidth. One QDR card running full tilt could easily flood the PCI slot, and that’s no good.

With PCI-Express 3.0, the raw bit rate is being jacked up to 8 GT/sec and the bandwidth per lane has been increased to 1 GB/sec. At the same time, PCI-Express Gen 3 moves to a more efficient 128b/130b encoding scheme to squeeze out some more bandwidth. So now an x8 slot can now handle 8 GB/sec of bandwidth; an x16 slot can handle 16 GB/sec of data. That means a QDR or FDR adapter card cannot flood the PCI-Express Gen 3 slot, and it also means there is a little headroom for future enhancements to InfiniBand and Ethernet, which currently runs at 1 or 10 gigabits per second in most networks and which is being upgraded now to 40 gigabits in some very expensive equipment that is really only useful for network backbones. But both InfiniBand and Ethernet are heading to 100 Gb/sec speeds, and that means that once again the network switch ports will be able to swamp a PCI bus slot unless PCI-Express Gen 4 can some to the rescue. Ditto for flash-based storage, which is quickly outpacing PCI slots with its I/O needs.

While the details of the Gen 4 spec are still being hammered out, the PCI-SIG says that they looked at 16 GT/sec and 24 GT/sec bit rates and that only the 16 GT/sec speed is possible with the current generation of chip technologies and using copper wires. To get beyond this to the higher 24 GT/sec speeds and higher will take optical links between PCI card and peripherals as well as a new style of PCI chips–and these are transitions that the system and peripheral industries are not yet prepared to make. In any event, a PCI-Express 4.0 x8 slot should weigh in at 16 GB/sec, or 128 gigabits not including the 1.5 percent overhead from the 128b/130b encoding that is being preserved with the move to Gen 4.

The PCI-SIG expects to finalize the PCI-Express 4.0 specification in the 2014 to 2015 time frame. IBM’s Power Systems machines are currently at PCI-Express Gen 2, and will likely be upgraded to Gen 3 with the Power7+ line of chips. But maybe IBM will wait for Power8. You never can tell with chip makers. It all depends on what customers need, and you can always count on them to do as little as possible to change a chip or system.