SPEC Launches Java Messaging Benchmark

Published: October 30, 2007

by Timothy Prickett Morgan

Getting a sense of the performance of applications on various server platforms has been made more problematic since the advent of distributed computing a decade ago. But real distributed computing--the kind driving Web-based applications today--is not just about separating application serving and database serving, which was complex enough. It is about having many different applications and pieces of middleware mashed up into a new kind of application that is inherently more complex than a simple two-tier or three-tier architecture.

To help company's better assess how iron performs in this more complicated hardware and software environment, the techies and vendors who participate in benchmark test development at the Standard Performance Evaluation Corporation have worked together to create the SPECjms2007 Java Message Service test, one of a suite of Java benchmarks. The new test does not, by the way, replace the SPECjAppServer2004 test, which was designed to stress Java-based application servers; nor does it obsolete the SPECjbb2005 server benchmark, which is an implementation of a Java-based online transaction environment.

According to SPEC, the SPECjms2007 test is the first industry-standard benchmark created to evaluate the performance of enterprise message-oriented middleware servers based on the Java Message Service. Message-oriented middleware is used in a number of financial services and telecommunications applications. (Remember all the noise that IBM made about MQ Series, now called WebSphere MQ, and Microsoft made about Message Queuing, or MSMQ, a few years back?) In any event, message queuing software is used to link distributed applications and the servers that support them together such that delays or broken links between machines do not stop transactions dead in their tracks; transactions are done by brokering message exchanges between machines and their software, and if a link gets broken or swamped, the messages are queued up for completion later. The net effect is that when you go to the ATM to get your money, you can get that money even if a part of the bank's financial applications are offline. Because of the flexibility that message queuing gives to applications, companies in the manufacturing, healthcare, and distribution industries are now looking at how to deploy this technology, and the JMS approach is one way to get this done in a Java environment. The benchmark spans the entire Java application, from hardware, the Java Virtual Machine and JMS service, the database, and the application that hammers on this software stack.

Unlike many benchmark tests that have been created by SPEC, the Transaction Processing Council, and other organizations, the SPECjms2007 test has two different metrics, which relates to the two ways in which JMS-style applications can be scaled--a horizontal and a vertical metric. The horizontal metric means that the workload on a network is scaled up by increasing the number of queue destinations that messages have to be transmitted to as part of the application; message traffic remains constant. The vertical metric scales the JMS application keeps the number of message destinations constant, but the number of messages is boosted. Both scenarios happen in the real world, hence the need to measure performance in both ways.

The Technische Universitat Darmstadt in Germany, the Apache project, and IT vendors BEA Systems, IBM, Oracle, and Sun Microsystems worked together to create the SPECjms2007 test.

No vendors have released any test results on the SPECjms2007 test as yet.

RELATED STORIES

SPEC Introduces MPI Supercomputer Benchmark

Power6-Based System i Performance: Your Mileage Will Vary

Power6-Based System i Performance and Bang for the Buck

Transaction Processing Council Launches TPC-E Benchmark

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot