Startup 3Leaf Systems Looks to Shake Up Server Virtualization

Published: May 1, 2007

by Timothy Prickett Morgan

A Silicon Valley startup with some prominent venture capital backers called 3Leaf Systems came out of stealth mode yesterday, and if the company's founders and engineers can accomplish their goals for changing the way server and storage virtualization are done on the X64 platform, they will not only shake up the market for virtualization products on the X64 platform. They might even give the X64 platform some competitive edges against RISC/Unix and mainframe alternatives.

Bob Quinn, the company's founder and chief executive officer, started 3Leaf Systems at the end of 2003 based on a handshake deal with Advanced Micro Devices based on the licensing of its HyperTransport system component interconnection technology. Like a number of other smart startups and some established IT players, Quinn saw an opportunity in the server space to make scale out systems--racks and racks of X64 servers--look and behave more like the trusty scale up systems--the mainframe, proprietary midrange, and RISC/Unix boxes that used to dominate data centers but which have been relegated to being database servers at most companies these days.

The issue is easy enough to explain: in the past decade, X86 and then X64 servers have been able to provide an order of magnitude of better bang for the buck than these other systems, which explains why they are popular. But the operating systems and the hardware that supports them on these X86 and X64 boxes did not have mainframe-class virtualization, and therefore at best X86 and X64 servers averaged maybe 10 percent utilization on average. If you pay 1/10th for a server but only use it at 1/10th of its capacity, you really are not paying less for processing, are you? Moreover, provisioning X86 and X64 servers can take days or weeks, depending on the setup, and scale up machines are, thanks to their engineering on both the hardware and software fronts (and in the interfaces where the two meet, which is where things usually go wrong), are very reliable. While X64 iron has improved in terms of reliability, so have RISC and mainframe processors, which are still inherently more reliable.

These issues contribute to the real and often hidden cost of X86 and X64 server sprawl, and these are not topics that IT managers and chief information officers want to bring up at meetings with their chief executive officers and bean counters.

3Leaf Systems wants to change all this with a set of hardware and software technologies called the Virtual Compute Environment, or VCE. "What VCE allows customers to do is construct a physical scale-out architecture, and then we let them logically deconstruct that scale-out to create a virtual scale-up server," says Quinn.

The company is going to start from the storage and work its way back into the servers with a set of virtualization products that are a little different from the software-based and hardware-assisted virtual machine and logical machine partitioning products that dominate the X64 server space today. 3Leaf Systems' VCE product is called the V-8000 Virtual I/O Server, and it tackles one of the trickiest virtualization issues today: breaking storage and networking from the physical hardware and associating it in a virtualized manner with operating systems. The V-8000 is an appliance that plugs into 20 X64 servers to virtualize the storage and networking I/O for them, and rather than thinking of it as a storage appliance, like some out of band storage virtualization products on the market, Quinn says that the V-8000 is a specialized server that extracts the state of storage from disk arrays and the physical servers and associates it with the operating systems running on servers, which is where it belongs. The V-8000 comes in a 2U form factor and has physical PCI-X Fibre Channel adapters from QLogic and Emulex and Gigabit Ethernet interfaces from Intel and Silicom. Disk storage hangs off of this box, and the links to storage are virtualized by the box, which looks like a normal SAN as far as Linux, Windows, or VMware's ESX Server hypervisor is concerned. The V-8000 has virtual HBAs and Gigabit Ethernet NICs that the servers see, quality of service features for shaping network traffic, and high availability clustering built into so two V-8000s can be put in a rack of servers to provide multipathing. The V-8000 links to servers through 10 Gigabit Ethernet fabrics or 10 Gigabit or 20 Gigabit InfiniBand links, which provide very bandwidth links between the servers and storage.

3Leaf Systems is charging $50,000 a pop for the V-8000, which can virtualize the storage of 20 X64 servers. Quinn says that even after pricing in the cost for the V-8000, it can reduce the capital expenditures on storage because each of those 20 servers do not have be equipped with an HBA for the SAN, and you don't have to provide redundant links for each server into the network switches. These adapters are expensive, and by using the V-8000 as a giant, shared storage and networking adapter, these costs can be eliminated and the network virtualized at the same time.

While the V-8000 is interesting, 3Leaf Systems' VCE product will get very interesting later this year when the second phase of its product comes to market. The company has developed a custom chip that will virtualize memory and compute resources within the server much as the V-8000 virtualized network links between server operating systems and physical SANs and network switches. This will be accomplished with a special chip, code-named "Aqua," that plugs into an Opteron chip socket. This chip, which talks to HyperTransport, will replace the backplane of an SMP or ccNUMA server that is normally implemented in the server motherboard. The Aqua chip extends HyperTransport out to the 10 Gigabit Ethernet or InfiniBand backplane and allows memory and processor cores in a cluster of servers to be exposed as a single resource pool--just like SMP and NUMA servers do.

Aqua has its own hypervisor, which allows NUMA clusters to be configured on the fly using memory spread across the servers in the rack; they can be scaled up and scaled down as Linux or Windows workloads demand. The operating systems can be further virtualized using VMware's ESX Server or XenSource's Xen hypervisors, if customers want to get sub-core partitioning or if they just like using these tools for whatever reason. But ESX Server and Xen are not required.

In a standard rack with 32 two-socket servers and two V-8000s, 3Leaf Systems can present a virtualized pool of 256 processor cores and 1 TB of main memory, which can be diced and sliced or glommed together into virtualized NUMA machines as workloads dictate. The VCE architecture can scale to 64,000 processor cores, according to Quinn.

The reason why this approach is possible now, according to Quinn, is that single-chip switches can be implemented that provide 320 GB/sec of bandwidth with latencies of under 100 nanoseconds. When this is couple with the fasted Ethernet or InfiniBand networking technologies, the combination can replace a backplane.

The Aqua chip and its related hypervisor and tools will be in beta testing at the end of 2007 and will be in production in the middle of 2008. 3Leaf Systems has 80 employees and has had three large customers beta testing its products since May 2006.

Incidentally, 3Leaf Systems doesn't expect companies to sacrifice an Opteron socket to make use of the VCE products, and expects that motherboard suppliers will add an extra socket to a server just for the Aqua chip. Thanks to AMD's Torenza open socket project, all of the specs to do so are available, and it is not that difficult for motherboard makers to add a third socket to a board.

While 3Leaf Systems is not saying much about its plans for the Xeon or Itanium server platforms, the fact that Intel is moving to Common System Interconnect structure very like HyperTransport with its future "Nehalem" chips in 2008 means that what 3Leaf Systems will accomplish with the Opteron-based server line this year will, in theory, be possible with Intel's products down the line. The fact that 3Leaf Systems just closed its second round of venture capital funding for $20 million with Intel Capital leading it off leads one to suspect this is exactly what is in the works. (The company had a $500,000 seed funding in June 2004 and had $12 million in first round venture capital in April 2005.)

RELATED STORIES

FastScale Takes a Different Approach to Virtualization and Provisioning

The X Factor: Virtualization Belongs in the System, Not in the Software

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot