A Little More Info on Red Hat Enterprise MRG

Published: July 15, 2008

by Timothy Prickett Morgan

Back in December 2007, commercial Linux distributor Red Hat announced its own entry in the realtime operating system space, throwing messaging and grid computing middleware into the mix for good measure. The product, called Red Hat Enterprise MRG (pronounced "merge"), is a collage of a number of different technologies, all of which are based on more modern variant of the Linux kernel that has been equipped to provide a consistent latency in transaction and message processing.

The first bits of Enterprise MRG--the realtime and messaging components--were announced at the Red Hat Summit a month ago in Boston, shipping in a 1.0 release; the grid components of the stack are due in a 1.1 update that comes out later this year.

Our December story, which you can read here, went into the technical details behind the Enterprise MRG product. I am not going to repeat all of that again here, since I said it right the first time. But still, the launch a few weeks ago of the Enterprise MRG 1.0 release left some questions unanswered, and so I sat down with Bryan Che, the product manager at Red Hat who is responsible for the MRG stack, to get a little more information.

First, let's talk about compatibility. Because of the way the new Linux 2.6.24.7 and the real-time kernel patch set known as CONFIG_PREEMPT_RT do not mess with the RHEL user space (the collection of kernel headers, glibc libraries, and such), that means any application that was compiled for RHEL 5.0, 5.1, or 5.2 (which are all based on the Linux 2.6.18 kernel) will run unchanged on the current Enterprise MRG 1.0 release and the future 1.1 release. This is also the same way that Red Hat maintained application compatibility inside RHEL 3 and RHEL 4, and it is also how rival Novell maintained compatibility inside its updates of SUSE Linux Enterprise Server 8, 9, and 10. The realtime patch set is something Red Hat has been working on in various ways for the past seven years, according to Che. In fact, Ingo Molnar, the systems programmer who is steering the CONFIG_PREEMPT_RT patch set at the kernel.org site where the Linux kernel lives, gets his paycheck from Red Hat.

Novell, IBM, Silicon Graphics, and a few others are also contributing to the realtime patch set as well, so don't think this is just Red Hat trying to establish a new standard for itself. This means of maintaining application compatibility stands in contrast with Novell's realtime Linux variant, SUSE Linux Enterprise Real Time, or SLERT 10, which includes extensions to the GNU GCC compiler set as well as kernel tweaks and therefore requires software developers and in-house coders recertify their SLES 10 applications on SLERT 10. What this means, of course, is that the process to port a RHEL 3 or RHEL 4 application stack to Enterprise MRG 1.0 is the same as moving to any of the RHEL 5.X releases. You don't have to do anything extra inside your application code to go realtime, access the messaging middleware, or grid up machinery for horizontal growth of computing capacity.

The one kind of compatibility that is not maintained, by the way, is hardware compatibility. The different kernel inside Enterprise MRG has different application programming interfaces and is thus far, like SLERT 10, only available on 32-bit X86 and 64-bit X64 iron. And that iron has to be recertified to support Enterprise MRG, just like hardware requires recertification for SLERT 10 because of similar kernel changes. Significantly, the customer base for realtime Linux applications--the typical financial services, telecommunications, industrial embedded controller, and military organizations--usually certifies a small number of machines, and they are used to this recertification process for other issues on modified server iron, such as is the case with NEBS compliance. Che says that given the initial customer set for Enterprise MRG, Red Hat was only asked to support X86 and X64 iron, but if there was demand, the company could provide variants to run on Power or Itanium processors, and even mainframes--all of which are supported by the generic RHEL 5.X stack.

Now, let's talk a little bit about performance. The issue with a realtime operating system is to make it behave in an absolutely predictable fashion. This is what all the kernel tweaking and system software tuning--such as in the network stack--is all about in the Enterprise MRG and SLERT releases. According to Che, the regularity of performance in Enterprise MRG, as measured in latency, is more important for many companies than reducing the absolute latency to the lowest levels. Check out this chart, and you can see instantly the different between RHEL 5.X and Enterprise MRG 1.0 running a financial trading application:

You see that thin green line down at the bottom of the chart? The real flat one? That's the response time for transactions in this application on a machine running Enterprise MRG. That red line shows how the response time increases at a steady rate until it reaches something akin to a steady state on RHEL 5, but the response times are all over the place. In a message-based system--like a stock trading system--delays mean lost money, so having predictable message and transaction response is a really big deal. So you can imagine how exciting that chart is to the nerd of a big brokerage house--if you can get him to worry about whether or not his employer is going to go bust this week, that is.

Now, as you well know, you can't get something for nothing in this world, so moving to a regular and predictable latency for response times comes at a cost: reduced throughput. On the financial, telco, and military workloads that Enterprise MRG has been tuned for (since these are the initial customers shelling out the dough), the throughput penalty has been a reduction of between 5 percent and 7 percent, according to Che. This seems pretty modest, and is in fact well within the normal headroom that big companies put into their systems as spare capacity for peak workloads. On some workloads, Che cautions, the throughput penalty can be a lot higher than this. Say, for instance, on applications that heavily rely on the file system, which Enterprise MRG has not been tuned for. (It can be, if customers need that and are willing to ask and pay for it, of course.)

The other thing to keep in mind when you consider Enterprise MRG is that many of the features that are going into this release will eventually--over the course of years--make their way into the standard Linux kernel, and Red Hat knows this and embraces the idea. Think of Enterprise MRG as something that sits between the Fedora development release and the commercial RHEL release and you get the right idea. For instance, the Completely Fair Scheduler that is part of the new Fedora 9 release's kernel that is now in Enterprise MRG will eventually be in the standard RHEL product. And by the time that happens, there will be other tweaks to the kernel that will differentiate Fedora and then Enterprise MRG from RHEL.

The other question I had about Enterprise MRG was about packaging and pricing. As it turns out, there are three components to the Enterprise MRG stack--realtime, messaging, and grid, as I explained back in December--but you can't but them all separately. You can buy the realtime add-ons separately, and you can buy the message queuing middleware and its tweaks separately. But the grid software will require both of these since it is tightly tied into both, according to Che. As for pricing, Red Hat is still not providing specifics for Enterprise MRG, but Che said that the realtime code will be available under a support contract that is less than twice as expensive as the standard RHEL product. (Novell charges $2,500 per server for the SLERT extensions to SLES 10, which costs $799 for a standard license.)

The messaging software, which will compete primarily with software from TIBCO in the financial services space, IBM's WebSphere MQ, a few other commercial products, and some homegrown code will be sold based on the number of processor sockets it runs on. The standard practice in the industry is to license proprietary messaging software and then charge around 20 percent maintenance fees per socket or processor core, and Che says that Red Hat will bring its charges for the messaging elements of Enterprise MRG down to be competitive with the per-socket pricing for maintenance on this products. This will be a huge cut in costs if customers can use Enterprise MRG instead of such products, much as Linux was a big saver compared to Unixes until Unix started to fight back. With the grid components only in technical preview, Che was not in the mood to talk about how the grid components might be priced, but again, you will have to buy the realtime and messaging extensions--or more precisely, but the support for them, since open source software is free--and then pay a premium atop that. My guess is the grid stuff will be priced on a per-node basis.

Enterprise MRG 1.0's realtime and messaging software is available now in North America and in Europe, and only in English. Later this year, when the 1.1 release is put out with the grid software, Enterprise MRG will be available globally and in the same languages that RHEL supports. The extensions can be run atop a tweaked version of the standard RHEL 5.X releases or the Advanced Platform releases, which add the Global File System (GFS) and unlimited Xen virtualization to the software for an extra amount of money on the support contract.

One last thing: Enterprise MRG does not run inside Xen hypervisors, and for now, because of the performance implications of virtualization, Red Hat's customers are not asking for it too.

RELATED STORIES

Red Hat's Profit Growth Stalls in Fiscal Q1, RHEL MRG Launched

Red Hat Breaks $500 Million in 2007, Aims 30 Percent Higher in 2008

Red Hat Releases Enterprise Linux 5.2 Beta

Red Hat Taps New CEO As It Reports Solid Third Quarter

Red Hat Goes Grid and Real Time with Enterprise MRG Distro

Red Hat to Use Automation, Virtualization to Eat the Server Space

The Low-Down on Red Hat Enterprise Linux 5.1 Enhancements

Red Hat, Reporting Q2, Reorganizes Operations for Growth

Red Hat, IBM Commit to Better Mainframe Linux

Red Hat Starts Fiscal 2008 with Modest Profit, Big Revenue Growth

Red Hat, IBM Commit to Better Mainframe Linux

RHEL 5: How's It Going?

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot