VMware Puts Virtualization to the Test with VMmark Benchmark

Published: July 24, 2007

by Timothy Prickett Morgan

Companies deploying server virtualization hypervisors to make their server infrastructure have a list of complaints, including the relatively high cost of hypervisors and the tools that make their use manageable by server administrators, the incompatibility of hypervisors, and the limited support for operating systems for certain hypervisors. They also complain that there are not independent tests to verify the performance of hypervisors running consolidated workloads. VMware wants to change that, and it has introduced a new benchmark test, called VMmark naturally, that it wants to become the industry standard.

According to Andrea Eubanks, senior director of enterprise and technical marketing at VMware, the VMmark test is the result of two years of engineering work by techies at VMware as well as collaboration with various benchmarking organizations, such as the Standard Performance Evaluation Corporation, or SPEC, which creates and administers a set of tests for gauging the performance of machines on various workloads. The VMmark is also the result of feedback from customers, who just want to know how many virtual machines they can cram onto a given box without having the whole thing thrash from overwork.

SPEC had already formed a subcommittee in October 2006 to look at a virtualization benchmark, in fact, which has the backing of Advanced Micro Devices, Dell, Fujitsu-Siemens, Hewlett-Packard, IBM, Intel, Microsoft, Red Hat, Sun Microsystems, Trigence, Unisys, and VMware. VMware didn't think it could wait any longer, so it launched VMmark ahead of whatever SPEC will ultimately do.

"It takes a long time to get a SPEC benchmark finalized," explained Eubanks, saying that even under the best of circumstances, it would take about a year to get a benchmark test through the workload studies, the detailed design, run rule creation, and initial testing before a virtualization benchmark could be released through SPEC.

VMware is not looking to control the benchmark, like SPEC and other benchmark certification organizations do, and it is happy to have its competitors use the tool to help customers do capacity planning and to make comparisons. But Eubanks cautions that VMmark was not created to assess the overhead of virtualization hypervisors, but rather to allow customers to get a sense of the number of typical infrastructure workloads they can put into VM slices on particular servers.

"It is not so important for VMware to publish results, but it is important for hardware vendors to do so and publish," says Eubanks. "And while Xen is remarkably absent from the SPEC subcommittee, we would love to have them join and we welcome their use of VMmark, too."

The VMmark test is available for download, free of charge. The VMmark test includes a mix of Windows and Linux workloads, which are run atop of a hypervisor. In this case, there are six workloads: an Exchange Server mail server on Windows; a customized version of the SPECjbb2005 Java transaction processing benchmark test running on Windows and using BEA Systems JRocket application server; an Apache Web server running on Red Hat Linux and being stressed by the SPECweb2005 benchmark; a database server running MySQL on Red Hat Linux running a test called DBhammer; the variant of the NetBench file server benchmark called dbench, which also runs on Red Hat Linux; and a homegrown standby server for the virtual machines, which doesn't eat much CPU at all, but which should be part of any test. With the VMmark test, you plunk down this stack of software, and then you keep adding VM partitions with the same software stack in six partitions--called a tile in the VMmark lingo--until the machine runs out of gas.

According to Eubanks, Dell has run tests on five of its PowerEdge servers, and has demonstrated that a four-socket PowerEdge 6950 server can host five tiles, for a total of 30 virtual machines.

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot