The Linux-Windows Warriors Get Better Weapons

by Timothy Prickett Morgan

You can crab all you want about vendor-sponsored research, such as the many studies that Microsoft has sponsored directly or endorsed as it tries to fight off the onslaught of Linux, but it does one very practical thing. The heated competition and even more heated words are, of course, amusing, but the truth is, the analytical weapons that the Windows allies and the Linux allies are deploying are getting more sophisticated. And this is a good thing for customers, even if it makes life difficult for commercial Linux distributors and Microsoft.

The latest interesting report in a volley of white papers coming out of the consultancy community at the behest of Microsoft comes from Security Innovation. Its report, called "Reliability: Analyzing Solution Uptime as Business Needs Change," is undeniably clever, which is one reason why Microsoft is paying for the study and touting it so much.

Even though Microsoft would never concede it--and certainly doesn't in this report--an awful lot of anecdotal evidence as well as empirical evidence has suggested that the core Linux operating system--including Linux 2.4, but particularly with Linux 2.6--is more reliable than Windows 2000 Server or Windows Server 2003. This latest study doesn't look at system uptime exclusively, but rather system uptime as it relates to a system that is not only being constantly patched but also a system that requires substantial updates to the applications that run on the servers running either Linux or Windows. This is, and everyone has to concede this, a scenario that is much closer to the reality of what IT organizations are facing. And in the Security Innovation study, Linux does not do as well as Windows.

Here's the scenario. Security Innovation hired three real Windows systems administrators and three real Linux administrators and simulated the kind of support and programming tasks they would be involved in over the course between July 2004 and June 2005 if they were maintaining a set of commercial applications that actually changed over time as the operating systems and middleware stacks underneath them also changed over time. The base Linux machine was Novell SUSE Linux Enterprise Server 8 with a third-party e-commerce application written in PHP and hosted on Apache with the MySQL database behind it; at the end of the year, the e-commerce site has to be upgraded with third-party extensions that allow personalized shopping experiences, allowing customers to look at their past buying, track shipping, and such. The base Windows machine has the same third-party software for e-commerce, and it runs atop Windows 2000 Server with Service Pack 4, equipped with the IIS Web server and backed by the SQL Server 2000 database; toward the end of the simulated year, the Windows box was upgraded to Windows Server 2003. Throughout the simulated year, patches to the operating system and middleware stack were done monthly, and every three months, a piece of the e-commerce system was upgraded. The first application upgrade was the addition of data mining software, the second was adding enhanced search capabilities to the e-commerce application, the third was adding customer lists management software that allowed targeted marketing to specific customers based on their buying habits, and the fourth was additional security plus the platform upgrade, to SUSE Linux Enterprise Server 9 and Windows Server 2003.

This is all very reasonable, and it is an intriguing scenario. But after I go over the results of the study, I have a few bones to pick with Security Innovations. They did some smart things, but they did some sly things, too, which I believe skewed the results. That said, many of the criticisms the report makes about open source software are, I believe, valid, and these are issues that the open source community would be wise to pay attention to.

According to the study, all three Windows administrators were able to do the 12 patches, four e-commerce application upgrades, and the platform upgrade in an average of 857 minutes (with administrators taking 772 minutes, 678 minutes, and 1,123 minutes individually). Each phase of the e-commerce upgrade had a stopgap of 600 minutes, and two of the Linux administrators went into the quicksand trying to add the enhanced search capabilities to the e-commerce application, eating up a huge amount of time. Perhaps significantly, each of the three Linux administrators had very different strategies for applying patches and functionality (namely a new glibc module), and ultimately one of the Linux admins ended up with a machine that would not boot because it had so many broken dependencies between software modules. Where to get software components is not always obvious, as anyone who has tried to build a Linux software stack for an application can attest to.

For instance, with the data mining component added in the first quarter of the simulated year, the Linux admins had to move from MySQL 3.23 to MySQL 4.1. SLES 8 didn't have MySQL 4.1, so they had to go to the MySQL site to get the code. But MySQL 4.1 used a more recent glibc compiler, which meant that had to be upgraded, too. Where do you get that? From SUSE, or from the open source GNU site? Which subrelease do you use? It can get very confusing--that much is certain--and determining whether or not your middleware is now supported by Novell or not can be as well. This is a very legitimate criticism, and it is precisely what makes an integrated software stack--like that available for IBM's AS/400 and iSeries servers for almost three decades now--and for the Microsoft Windows platform since Windows Server 2000 SP really became a stable, usable platform--better than a hodge-podge of very innovative but often not very integrated components.

Resolving dependencies in the Linux software stack is a big pain in the neck. And the Yast systems management tool that was improved in SLES 9, but it is still not flawless as far as our own use of it here at IT Jungle goes and, apparently, among the three Linux administrators. Of course, what Yast is trying to do is very complex, and is not necessary with the Windows stack. Microsoft makes sure the dependencies are in line for its own software, and third parties have to worry about how they hew to the Windows stack if they want to be in business. The people who run open source projects are worrying about adding features and improving their code--they are not worrying as much about all of the possible dependencies in a particular Linux instance. This is why Red Hat, Novell, Mandriva, and others can charge for this Linux distros. We could debate for a long time how well or poorly the commercial Linux distributors do this.

Anyway, in the analysis done by Security Innovations, the Windows platform had 39 cumulative patches prior to the upgrade to Windows Server 2003 and no dependency failures among the applications in the stack, while Linux had 187 cumulative patches prior to the upgrade to SLES 9 and 14 cumulative dependency failures among applications. And, most tellingly, it took the three Linux administrators 41 percent longer than the Windows administrators to get their systems patched and upgraded.

That's not a very good showing for Linux, obviously.

But before we take this all to heart, it is time for bone picking. The first question I have is this: Why should I trust a company that is being paid by Microsoft to pick the so-called Linux experts? If you really wanted to do the Worldwide System Admin Smackdown properly, you would have Microsoft pick three admins and a Linux vendor such as Novell pick three admins, and name them and show their qualifications. You might even want to supply them with an occasional metal chair.

The comparison also ignores the inherently better security and reliability of a Linux or Unix system--at least according to many server buyers and an awful lot of other studies and anecdotal evidence--compared to Windows. Yes, the Linux stack is less integrated, but it has other benefits. Also, why does anyone assume that you want to get MySQL support from Novell just because Novell offers it? If you are building a software stack, you get SUSE to support Linux, MySQL to support MySQL, and so forth. Yes, this is more complex, but you don't limit your support to the options that come in a base operating system. To be more specific, if you are installing an e-commerce application in July 2004, you do not install it on MySQL 3, but rather MySQL 4.1, which was the then-current release from MySQL. I would love to see how well the new SQL Server 2005 did on a similar comparison. To put it bluntly, not much changed in the software stack on the Microsoft side, and a lot was changed on the Linux side. So it is no wonder that the Linux admins had to cope with more. The move from SLES 8 to SLES 9 is akin to the move from Windows Server 2003 to the future "Longhorn" Vista Server, and the jump from MySQL 3 to 4.1 is a pretty big jump, too, as is the jump from MySQL 4.1 to the new MySQL 5.

If you really wanted to make a fair comparison--if there is such a thing--you would put the same exact database on both the Windows platforms, such as Oracle 10g or IBM, and the same exact middleware stack, and see how that worked out. You could also do some interesting variations, such as making Windows admins do Linux and Linux admins do Windows, and find reasonably skilled admins who have no experience with either and see how they did with the same software.

Finally, without knowing what the e-commerce application stack was, it is very hard to say how much this comparison was skewed. Just because something is popularly used by the Fortune 500 does not mean it runs well on either Linux or Windows. I have this sneaking suspicion that the code Security Innovation chose to test on its Linux and Windows boxes is a set of Windows apps that was recently ported to the LAMP stack. And as we all know, there is a big difference between a mature implementation and a new one, even if it is supposed to be the same exact code. By not saying what this key software was, the comparison is called into question.

Suffice it to say, I think that Security Innovation has brought up a very interesting point, and an intriguing methodology. But it needs to be made a little more transparent about the details and a lot more fair in its comparison.

And I would be happy to supply a dozen metal chairs, spandex tights, and feather boas if someone wants to do this right.

RELATED RESOURCES

Reliability: Analyzing Solution Uptime as Business Needs Change (Security Innovation, November 2005)

Role Comparison Security Report--Database Server Role (Security Innovation, June 2005)