Fabric7 Creates Flexible Opteron Server for Linux, Windows
by Timothy Prickett Morgan
After years and years of consolidation in the server industry, where a cornucopia of vendors selling a wide variety of server architectures has been crunched down by market forces into a relative handful of server makers selling only a few different server architectures and only supporting a number of operating system platforms, it is refreshing to see a new vendor like Fabric7 Systems enter the market with an interesting new architecture on the Opteron processor from AMD.
Fabric7 was founded by a bunch of server experts who hail from Sun Microsystems and the former Sequent Computer Systems (which was acquired by IBM in July 1999 for $810 million). Fabric7 was founded in May 2002 and has been operating in stealth mode in Silicon Valley--in Mountain View, Calif., to be specific--putting together a slightly new kind of server design. This design creates a fabric made out of virtualized commodity components, including Opteron processor nodes, Gigabit Ethernet networking nodes to link to the outside world, Fibre Channel connectivity nodes to link to storage arrays, and specialized processing nodes that include TCP/IP, XML, SSL, and SLB (server load balancing) acceleration. Each of these nodes can be virtualized--meaning, carved up into logical pieces and cross-connected to each other through a high-speed, low-latency crossbar switch. This crossbar, according to Sharad Mehrotra, who is one of the founders of Fabric7 and who is now its president and CEO, is based on a chipset used in high-performance routers and switches--the IBM Prizma chipset, to be specific, which was created by IBM Research in the late 1990s and sold to chip partner AMCC. You have to be careful what you discard in the IT business, because someone can always take what you didn't think was working right and make something surprisingly good out of it.
The Fabric7 Q80 and Q160 servers as well as the executives who have founded the company are an excellent example of how the IT ecosystem likes to conserve and reuse ideas. Mehrotra founded a high-end networking startup in 1999 called Procket Networks, which raised $272 million in venture capital and which eventually sold itself to Cisco Systems in June 2004 for $89 million. Before founding Procket, Mehrotra was one of the designers of the ill-fated "Millennium" UltraSparc-IV processor at Sun Microsystems after putting in a few years as a staff engineer. His two other co-founders are Nakul Saraiya, who is Fabric7's chief software architect, and Thomas Lovett, chief server architect. Prior to founding Fabric7, Saraiya was the principal engineer at Akamai Technologies, and was also a senior staff engineer at Sun who worked on Java programming and Sparc server SMP scalability. Lovett heads up the Fabric7 server development team from Portland, Oregon, which stands to reason since he was the chief architect of Sequent's NUMA-Q scalable X86 servers. Lovett moved over to IBM and worked on the "Summit" xSeries servers, which are in essence the kicker to the NUMA-Qs.
When these three guys started up a server company in May 2002, you might think that you'd have to be crazy to invest in a new server company, especially one that was betting the farm on the Opteron processors. But that is not what the venture capitalists seem to think, particularly based on the skill and experience of the founders. So far, Fabric7 has been able to raise $32 million in private funding from New Enterprise Associates, Goldman Sachs, Selby Venture Partners, Vanguard Ventures, Foundation Capital, Sanmina-SCI, and Yasuda Enterprise Development.
There are a few secrets to the Fabric7 Q series of servers, and one of them is to weave high performance but commodity components together to make a very powerful yet inexpensive server platform. The other secret is that the design of the Q machines is that rather than being a server design that adds on storage connectivity, networking, and server function acceleration as afterthoughts far out on the I/O buses, these features are right in the heart of the system on that high-speed crossbar that connects computing elements together. This is the fabric part of Fabric7, and if you remember the promises that were made about the InfiniBand switched fabric architecture that were made many years ago by IBM, Intel, and other InfiniBand backers, this will all sound very familiar. InfiniBand was supposed to be a switched fabric that would end up being the connectivity between server nodes and storage nodes, essentially becoming the backplane in a server, and with very high bandwidth and low latency on those InfiniBand links, the server nodes were supposed to be able to function like a cluster of small machines or a giant NUMA-oid SMP server. In plain English, the same backplane that does the SMP scaling is used to talk to the outside world through virtualized Ethernet and Fibre Channel links.
The server nodes in the Q80 and Q160 servers are based on homemade two-socket Opteron motherboards that were designed by Fabric7 and that are made by Sanmina-SCI. The machines can use either single- or dual-core Opteron 200 Series or 800 Series processors, and the homegrown Q-Par partitioning software sold with the servers can carve the machine up into partitions with two-socket granularity. The server boards use registered ECC DDR1 main memory running at 133 MHz, 166 MHz, or 200 MHz, with four memory slots per processor or a total of 16 slots per board. The Q servers support 1 GB, 2 GB, and 4 GB DIMMs, and have hardware memory scrubbing and chipkill. The server offload boards that handle TCP/IP, SSL, XML, and SLB acceleration are based on two-socket Opteron processors, and the system controller is also an Opteron card. This system controller runs the Q-Par partitioning software and the related out-of-band system management software, which is called Q-Visor. The system controller also has an ARM-based diagnostics processor. Because the switch is built into the server architecture, if you want to add network bandwidth, storage bandwidth, or other features, you just plug them in and they are ready to be activated. The Q-Visor program has both graphical and command-line modes, and it allows administrators to build virtual LAN and Fibre Channel connections.
The big box in the Q line of servers from Frabric7 is the Q160, which has 14 server sockets, which means it can span up to either 14 or 28 Opteron cores in a single frame. The machine can support up to seven hard Q-Par partitions, and supports partition sizes of two, four, eight, and 16 cores in a single system image. Virtualization software such as VMware's ESX Server or the open source Xen hypervisor can be used within these Q-Pars to further slice up the machine. The Q-Pars can be changed dynamically. Using 2 GB DIMMs, the Q160 has a top memory of 128 GB. The switched I/O in that crossbar has 128 Gb/sec of non-blocking I/O, and it provides 40 Gb/sec of bandwidth between the processor and memory complexes. The Q160 can support up to 112 hardware-accelerated virtual Gigabit Ethernet NICs or virtual Fibre Channel host bus adapters, with 30 Gb/sec of full-duplex line rate I/O capacity, which can be used to build out larger clusters of machines. Mehrotra says that several tens of the Q160s can be meshed together, and that the 16-core scalability of the single system image will be extended with the next generation of Opterons. (Hint, hint. The Opteron Rev Fs will have more HyperTransport links, and faster ones, too, is my guess.)
"We have much better true I/O than our Unix counterparts," explains Mehrotra. "I do not think it is useful to be counting I/O slots." In fact, he says that unlike big Unix boxes, which have less aggregate I/O capability on their crossbars than if you add up all the capacity of each I/O slot running at full speed, the Fabric7 design goes in the opposite direction, offering a lot more crossbar bandwidth than the I/O slots can pull.
The company has also created a smaller box called the Q80, which has four two-socket motherboards and can support partitions with 2, 4, 8, or 16 cores. This box has a smaller chassis, and supports up to eight Opteron processors--again, single- or dual-core 800 Series processors. The Q80 supports up to 64 GB of main memory using 2 GB DIMMs and offers industry-standard I/O, including eight Gigabit Ethernet ports and eight PCI-X slots.
The Q160 is shipping now, and it is being aimed exclusively at Linux customers. It supports (Red Hat Enterprise Linux 4 and Novell SUSE Linux Enterprise Server 9. The Q80 will begin trials in January, and will support both Linux (Red Hat and Novell again) as well as Windows Server 2003. Considering that Windows supports 64-core scalability right now, it will also support 16-core SMP scalability on the Q160 when it is eventually moved that that machine. Whether customers pick Linux or Windows, it runs off-the-shelf versions of either operating system with no changes--a design feature of the Q series machines. Fabric7 is not yet thinking about Solaris 10, which could technically be supported on the servers, and it is absolutely uninterested in Intel's Xeon or Itanium processors.
The base Q80 machine with a few Opteron cores, 16 GB of main memory, and the basic OS boot drives will sell for $40,000. A base Q160 with redundant power, one crossbar, four CPU sockets, and 16 GB of memory will list for $140,000. A heavily configured machine with 16 cores and 128 GB of main memory--which Mehrotra says "will be very dangerous to a Sun Fire 6900, an IBM p5 570, or half of an HP Superdome"--will cost around $400,000. He expects the average selling price of the Q160 to be around $250,000.