What Does Microsoft's Latest Windows-Versus-Linux Test Show?

by Alex Woodie

Microsoft continues to add more material to its 'Get the Facts' Web site to bolster the case for Windows against Linux. The latest addition, which was posted last week, describes a series of tests that were structured to determine which production X86-server environment was easier to set up and run--the one running Windows Server 2003, or the one running Red Hat Enterprise Linux 3. We'll give you one try to guess who came out on top. (Here's a hint: It wasn't the Linux.)

The latest series of tests pitting Windows against Linux were performed by VeriTest, which is the testing division of Lionbridge Technologies. VeriTest is no newbie to the testing game, and has been contracted by many tech companies, including Oracle and Sun Microsystems, to set up and carry out certification tests. Novell contracted with VeriTest earlier this year to perform certification tests for products running on SuSE Linux.

Microsoft paid VeriTest to compare the usability and ease-of-use of Windows Server 2003 against Red Hat Enterprise Linux 3, and when you pay for something, you have a certain amount of say over what those tests will measure and, therefore, some degree of control over how the tests will play out. (The way these tests work, Microsoft wouldn't have released the test results to the public, unless they reflected positively on Windows.) Because of that, the latest series of tests should be assessed with a skeptical attitude, one that is even more suspicious than the Yankee Group opinion poll survey results we reviewed last week, because Yankee Group is recording the responses of outside sources and was not paid by any company to collect the data. VeriTest, which was paid by Microsoft to carry out this test (and many others in the past in which Microsoft ranked favorably against, mostly, Linux), is testing exactly what Microsoft wants tested.

With that said, the detailed and multi-faceted real-world tests that VeriTest set up and executed do provide some insight into the different approaches that Linux and Windows administrators take in day-to-day operations. Windows is more feature-rich than Linux in many ways, with more stable drivers available and a standard GUI, and therefore one might expect there to be fewer compatibility or complex configuration issues with Windows. Linux administrators, on the other hand, typically take a more a piece-meal approach to building, configuring, and integrating their basic infrastructure components, and therefore one might expect Linux administrators to need to be more creative in finding solutions to compatibility problems than their Windows counterparts, who typically only have one path set out for them. These different approaches are evident in the VeriTest study.

Test Results

In this test, VeriTest attempted to simulate a medium sized business environment with about 200 employees that uses three Hewlett-Packard ProLiant DL380 G3 servers that are running pretty standard e-mail, file/print, and infrastructure server workloads, and which are set-up in failure-prone states. The Windows system was running Windows Server 2003, Exchange Server 2003, DNS, and DHCP services, while the Linux system was running Red Hat Enterprise Linux 3, OpenLDAP (which was troublesome for most Linux admins), DNS, DHCP, Sendmail, and Samba.

The tests measured how long it took 36 experienced (but not too experienced) Red Hat Linux and Windows Server 2003 administrators (who went through a pre-screening process) to complete "proactive" tasks, such as updating and reconfiguring their un-patched and unprotected systems to make them more secure and reliable, and how quickly they could complete "reactive" tasks, which involved troubleshooting and recovering from some curveballs, such as device or system failures, that VeriTest threw their way.

Windows Server 2003 won the reactive part of the test, with an average downtime of about 4 hours and 20 minutes, while Red Hat Enterprise 3 was down an average of about 5 hours. Windows also won the proactive part of the test, with administrators completing tasks in about 14 hours and 24 minutes, while it took the Linux administrators an average of 22 hours and 48 minutes to complete proactive tasks. VeriTest then added those two numbers together, and the Windows administrators' average score was about 18 hours and 44 minutes, while Linux administrators required a whopping 27 hours and 48 minutes to complete the proactive and reactive tasks--mostly due to those proactive tasks. (Those proactive task numbers for Linux are so high, we're wondering if some of those open-source enthusiasts took it upon themselves to write their own drivers.)

VeriTest also measured the accuracy with which the Windows and Linux administrators completed both sets of tasks, and Windows, once again, came out on top, with a total of 280 correctly completed tasks, compared to 248 correctly completed tasks for the Linux admins.

Caveats

The Linux administrators struggled with OpenLDAP on Red Hat Enterprise Linux, with only two of the 18 Linux admins successfully completing the OpenLDAP task, compared with a majority of Windows administrators, who used ActiveDirectory, which is built into the OS.

"We believe a primary factor for the disparity in these results is the complexity of managing OpenLDAP on Red Hat Enterprise Linux AS 3.0 compared to the relative simplicity of managing Active Directory on Windows Server 2003," VeriTest said in its 62-page report (which is available for download here). The Linux "administrators chose several different OpenLDAP management tools. The most widely used tool was a GUI browser-based tool called Webmin. Other tools used included LDAP Account Manager, LDAP Browser/Editor, and the Idealx SMB-LDAP command line tools installed on the systems by VeriTest." Those using Idealx SMB-LDAP had the greatest success completing the proactive directory-related tasks.

Interestingly, other areas where the Linux administrators struggled were configuring the HP LaserJet 2300 printers and the HP DAT 72 tape drives, with the Linux people spending about 3 hours and 45 minutes on these tasks, compared to an average of just 1 hour and 25 minutes for the Windows guys.

"One [Linux administrator] mentioned specifically that configuring the tape drive was hard to understand and that he didn't like having to create scripts to enable the drive," VeriTest writes in its report. "In general, the device driver installation process on Red Hat Enterprise Linux AS 3.0 is not as consistent as the binary driver installation packages for Windows Server 2003, resulting in more diversity in the steps required to install and configure devices…. No Windows Server 2003 administrators had any issues with tape drive or printer configuration."

In a time trial like this, the diversity of different solutions available in the open-source Linux world actually hurts productivity, although it might be beneficial to solve non-standard problems in the long run. While Unix operating systems are renowned for their flexibility, this diversity of solutions necessitates a certain level of built-in complexity, which is evident in this test of short-term administrative prowess and trouble-shooting abilities. In many cases, there's only one way to do things in Windows, which is definitely an advantage to administrators in a test like this. Of course, having only one way to do things, one solution for the problem--the Microsoft way--can have it's disadvantages, but that's another test for another day.

Microsoft also added a Q&A with a vice president at VeriTest, Katrina Teague, to its PressPass page. To read the Q&A, in which Teague discusses VeriTest's testing methodologies and procedures, see "Testing the Reliability of Microsoft Windows Server 2003".