Distros Pick Apart Forrester Linux-Windows Security Rankings

by Alex Woodie

Not surprisingly, the major Linux distributors are unhappy with Forrester Research for its conclusion that Windows is more secure than Linux. In a joint statement, Debian, Mandrake, Red Hat, and SuSE (which is now owned by Novell) criticized the analyst group for treating "all vulnerabilities as equal, regardless of their risk to users." As a result, they say, less important security holes were given the same weight as bona fide high risk vulnerabilities in Forrester's assessment, pulling down Linux's security batting average compared to Windows.

In late March, Forrester published the results of its study, in which it tried to quantify how Windows compares to Linux in terms of the number and frequency of publicly reported high-severity vulnerabilities between June 2002 and May 2003, and the time it took Microsoft and the open source community (in conjunction with the commercial Linux distributors) to make patches available for those vulnerabilities (see "Forrester Says Windows Is More Secure Than Linux").

Forrester concluded that, for Red Hat, it took an average of 57 days to publish fixes for 128 high-severity security flaws, once those flaws were disclosed to the open-source community (what Forrester called "all days of risk" in its study). Forrester also calculated that Red Hat had 47 days of "distribution days of risk," which meant that Red Hat issued a fix 47 days after the open source community made a general fix available. Debian had an "all days risk" average of 57 days to publish fixes for 163 high-severity security flaws, and a "distribution days of risk" of 32. Mandrake, had an average of 82 days of "all days risk" to publish fixes for 120 high-severity security flaws, while its "distribution days risk" average was 56. SuSE took an average of 74 "all days of risk" to publish fixes for 176 high-severity security flaws, and had a "distribution days risk" average of 54. Finally, for Microsoft, it took an average of 25 days for the company to publish fixes for its 86 high-severity security flaws.

Since the Linux distributors work with the developers of the open source Linux kernel, they can only customize and deliver fixes to users of their particular distribution after the open source community has documented the flaws and provided a fix. Since Microsoft keeps all development in-house, it doesn't have to worry about this extra step, and Forrester's results bear this out. However, this is one of the points that the Linux distributors are upset, and maybe rightly so.

The Linux distributors say that Forrester's technique "erroneously treats all vulnerabilities as equal, regardless of the risk they pose," when, in fact, all vulnerabilities are not equal. The distributors say that, based on the information exchanges they have with security research organizations, such as CERT, the British National Infrastructure Security Co-Ordination Centre, National Institute of Standards and Technology, they work to evaluate the severity of each vulnerability, and then prioritize each vulnerability based on its severity. As a result, important flaws get fixed within hours, while fixes for less important issues will be delayed.

"Not all vulnerabilities have an equal impact on all users," the Linux distributors say in their joint statement, which is available here. "An attempt has been made to allocate a severity to vulnerabilities using data from a third party. [H]owever the classification of 'high-severity vulnerabilities is not sufficient: The mere announcement of a vulnerability by a particular security organization does not necessarily make the vulnerability severe. [S]imilarly, the ability to exploit a weakness over the network (remote) is often irrelevant to the vulnerability's severity."

"We believe the report does not treat vendors of Free Software and the single closed source vendor in the same way," the distributors continue. "The openness, transparency, and traceability of the source code is added value in addition to the larger variety of software packages available. Finally, the claim that one software vendor had fixed 100 percent of their flaws during the period of the report should be incentive for a closer investigation of the conclusions the report presents."