The Four Hundred

It is significantly less expensive to implement and run ERP applications on OS/400 than on other popular server operating system environments, according to a new study by META Group. The study found that, compared with installing and running ERP applications on the Windows NT, Windows 2000, AIX, HP-UX, and Solaris platforms, OS/400-based ERP costs anywhere from two to five times less over a one-year period, and seven times less over a three-year period. But is it accurate?

The report, titled The Impact of OS/Platform Selection on the Cost of ERP Implementation, Use, and Management, is a detailed analysis of data gathered from 195 U.S. companies--mostly manufacturers--in the first half of 2001. The report summarizes ERP-related costs incurred over the first year of implementation and also after three years of implementation. META categorized each company according to the operating system and server that runs its ERP software, and then tabulated expenses related to hardware, hardware maintenance, OS license, OS maintenance and warranty, ERP software, ERP maintenance, and database costs. Instead of gathering actual data for personnel-related costs, META assigned a fixed cost of $78,000 per employee, per year, to the data from each company, and multiplied that by the number of IT staff employed. In seeking to get a clear view of just the effect that a platform decision has on a host-based ERP implementation, META did not gather cost data relating to client workstations, networking, or high availability setups.

The report, which was authored by Kristopher Hanna, a META consulting director, reflects more than 2,000 hours of work by senior META analysts and consultants over a period of months. META solicited participants in the survey by sending out e-mail invitations to a list of more than 12,000 companies that the group had gathered on its own. The initial collection of about 450 respondents was cut by nearly 30 percent because they had complex ERP implementations, most often consisting of a front-end Windows application server connected to a back-end OS/400 or Unix database server. META did not invite these companies to participate in the second-phase of the survey because the group felt it would jeopardize the goal of the study, which was to isolate the effect of platform choice on cost of ownership.

Three different companies commissioned the study, but the Stamford, Connecticut, consulting and analyst firm declined to identify those companies. However, the executive summary of the report was distributed to members of the press by IBM's public relations firm, which points toward IBM as being one of the companies that commissioned the report. IBM has not publicized the fact it did not commission the report.

For years, it has been an accepted fact in the AS/400 community that IT systems based on the OS/400 platform cost less than similar systems installed on other platforms when the costs are spread out over a period of years. Yes, the upfront outlay of cash to obtain iSeries hardware is often higher than other platforms, especially compared with Microsoft Windows operating systems running on Intel-based hardware. But when you factor in the cost of IT staff, the fact that OS/400 comes bundled with a database management system, and you spread those costs out over three to five years, the prevailing wisdom in the community says you are actually getting a better deal with OS/400 than with other platforms. IBM's iSeries managers have told us as much on numerous occasions.

IDC says so, too. Last year, the prestigious Boston IT research firm published a white paper called Server Cost of Ownership in ERM Customer Sites that found OS/400-based enterprise resource management implementations cost anywhere from 45 to 95 percent less than comparable Unix or Intel-based solutions over three- and five-year periods, in large and small customers sites alike. That report is accessible from the iSeries Nation Web site.

Now, the new META Group study claims the cost advantages of OS/400 are significantly better than the savings IDC would have you believe, and you don't have to wait three to five years to get them. In fact, with OS/400, you will pay half the cost of a comparable Windows-NT-based ERP implementation, and that's in the first year! And if you spread the costs out over three years, a Solaris-based ERP setup will run you seven times the cost of a comparable OS/400-based system. That level of savings makes you wonder why any company would buy a Solaris system when it could run ERP on OS/400.

After the first year, according to META, the total cost to run an OS/400-ERP system would be, on average, a little less than $40,000. In its summary cost comparison, META found Windows 2000, Windows NT, and AIX cost about twice what an OS/400 setup would cost; HP-UX cost about three times as much; and Solaris cost five times more. After the third year, according to the META report, the typical respondent had spent, on average, a little more than $50,000 to run an OS/400-based ERP system. By comparison, Windows NT cost about 20 percent more, Windows 2000 cost two times as much, AIX was three times higher, HP-UX was five times higher, and Solaris was seven times higher.

So how did META arrive at these fantastic (from an OS/400 point of view) cost of ownership figures? Let's take a closer look at META's methodology and the numbers they arrived at.

First of all, the total-cost-of-ownership comparisons META amassed (which are summarized above) are based on the number of concurrent users. The information META collected relating to the cost of hardware, software, and personnel was normalized against the concurrent user figure, which basically means that META took the total amount of money each company said they spent supporting the ERP application on the host system and then divided that total by the number of concurrent users the respondents said used the ERP application at any given time. Then META averaged the numbers to arrive at the figures used in the cost comparison.

META used its own judgment to determine the number of concurrent users at each company. To arrive at this crucial figure, on which all of the cost comparisons are based, META asked each company how many seats they purchased from their ERP vendor and the average number of users they had signed on to the host at any given time. Hanna, the report's author, said META would then arrive at its final concurrent-user figure based on the consistent value each user provided to the organization.

In looking at the findings, you should also realize that META's cost comparisons do not include information from companies whose ERP systems support fewer than 20 people. META found that the "costs for ERP implementation with 20 or less users across all O/S platforms were highly variable," so the group decided to exclude this data from their findings. It is unclear how many of the initial 195 companies fell into this category, but a safe guess would be 86, considering that the cost comparison section of the report reflects the data gathered from 109 companies. It is unknown how the inclusion of data from companies with 20 or fewer users would have altered the cost comparison because META did not provide that information in its report. It is quite likely that most of these smaller shops were running Windows-based ERP systems, which tend to be smaller in scale than Unix or OS/400 implementations. If these assumptions are correct, then the exclusion of this data would make META's cost-comparison findings more relevant to higher-end Windows implementations than to the typical Windows implementation.

Another item of note is that the number of respondents for each platform varied quite a bit. While the cost-comparison findings reflect data gathered from 34 HP-UX shops and 25 Windows NT shops, the study only included three Windows 2000 shops and nine AIX shops, hardly what one would consider a wide, or even consistent, cross-section. META also did not conduct statistical analyses to determine the margin of error of its findings. There is a simple equation to calculate the margin of error of a random sample from a general population. It's 1 divided by the square root of your random sample. When you plot this equation on a graph, it gives you an exponential curve, so that a sample size of 10 corresponds to an error rate of 31 percent; a sample size of 20 gives you a 22 percent error rate, a sample size of 50 corresponds to 14 percent, and a sample size of 100 gives you an error of about 10 percent. You can never completely eliminate the margin of error in a survey of a statically random sample, but you can see why the sample size must number in the several hundreds to get the error rate below a generally acceptable level of 5 percent. Whether or not this standard error calculation can be cleanly applied to META's numbers is a subject of mathematical debate, but it does raise some questions as to the precision of META's findings. For the record, META's cost comparisons reflected data gathered from 18 OS/400 shops and 20 Solaris shops.

The size of the companies surveyed also varied quite a bit, and there was no control factor built into the study to enable one to compare the ERP-related costs of companies that are similar in size. The study states that the size of the companies ranged from less than $10 million to $43 billion in annual revenue, and that 50 percent of the companies had revenues ranging from $500 million to $2 billion.

Likewise, there was no control factor built in to the study that would allow you to compare the costs of implementing the same ERP application on different platforms. Hanna said META encountered familiar big-name ERP vendors in the study, such as SAP, Oracle, PeopleSoft, J.D. Edwards, and QAD. Many of the companies had altered their ERP software to some degree, Hanna said, but there was no category for companies that built their own system.

When asked if he was surprised that OS/400 faired so well against other platforms, Hanna responded that he did not have an opinion. Hanna, a former IBMer who worked in Rochester, Minnesota, for three years, said he respects the loyalty that OS/400 users have for the platform.

At first glance, META's report would seem to confirm what many in the industry have suspected for some time, that the ease-of-use, stability, and integrated nature of the OS/400 platform have had positive, tangible impacts on the finances of companies that have chosen to use it to run their critical business applications. In setting out to demonstrate the effect that a decision to invest in a specific operating system and server platform has on companies, META declared that the common denominator would be the existence of an ERP system at a customer site, and the key differentiating factor would be the number of concurrent users.

However, META's study doesn't live up to expectations in several key areas. First, the study doesn't take into account the wide variety of different types of ERP systems in use at U.S. companies or the prevalence of companies to invest in ERP applications that provide functionality consistent with their overall size and budget. Is it fair to compare the cost of a mature accounting system written in RPG III 15 years ago with a functionally superior mysap.com application from SAP or with J.D. Edwards' latest OneWorld software? Applications written in RPG--the language in which most OS/400 ERP application are written--can be optimized to run very fast on OS/400 and will consume less computing resources per person for the simple fact that they do not typically have a graphical user interface. Additionally, many ERP vendors with older RPG applications have stopped significantly enhancing them because the demand for them is not what it once was, which will have the effect of reducing the initial ERP license cost and the cost of continuing maintenance for an RPG application compared with a five-year-old ERP application written in C++ or Java.

Second, the sample sizes are simply not large enough to produce meaningful benchmark averages. This raises additional concerns because META collected ERP-related cost information from individuals who weren't in charge of those areas of their companies. Granted, there's probably no good way to gather hard cost data, short of conducting full-scale audits. Nonetheless, one should be aware of this fudge-factor and its potential effect on small sample sizes. Additionally, META said that more than half of the companies that participated in the study refused to provide the key cost data.

Lastly, META's definition of concurrent users (as those providing consistent value to an organization) leaves too much room for interpretation, especially when considering it is META's defining factor in its cost comparisons. What is "consistent value"? How does it relate to the number of people who are signed on to a server at any one time?

META's study indicates that a huge gap exists between the cost of OS/400-based ERP implementations and the cost of running ERP applications on other platforms. Most people familiar with the AS/400 and iSeries would probably believe that the OS/400 platform gives some degree of cost savings over the long run, but would point to any number of other factors, such as its stability and ease of use, as other important advantages. And if iSeries servers are such a bargain, why did IBM just cut the price of its entry-level servers by 50 percent?

Most platform veterans would probably never in their wildest dreams think it's seven times more expensive to run ERP software on Solaris than on OS/400. Common sense says that when something seems too good to be true, it usually is.