Evans Data Ranks Integrated Development Environments

May 27, 2008 Timothy Prickett Morgan

Farmers have their tractors. Managers have their PowerPoint and Excel programs. Plumbers have their wrenches and torches. And programmers have something given the utterly inelegant name of the integrated development environment, which just goes to show you that the marketing people get to name things far too often. Call it what you will, but the IDE is the tool that most of the professional programmers in the world–and there are millions and millions of them–stare at each day as they craft the applications that let the rest of us do our work.

Picking the right IDE is not an easy task, presumably, and programmers have their own preferences and prejudices regarding programming languages–just like the rest of us do for the important tools in our work lives–so it should come as no surprise that they would have strong feelings regarding the IDE that integrates and consolidates the programming tools that they use in their work. To help give programmers a sense of what’s hot and what’s not in the IDE arena, Evans Data, the consulting company that focuses on all manner of issues relating to programming and programmers, has released a study of IDEs that is based on a survey of 1,200 developers around the world that had users of IDEs from seven key suppliers rate them based on 15 different metrics, including ease of use, compiler performance, performance of resulting apps, quality of tech support, documentation, debugger, editor, and application modeling tools; programmers were also asked to give each attribute a relative importance ranking, too, and the whole batch of data was run through the statistical equivalent of a food processor to rank the IDEs relative to each other. Evans Data ran the survey behind the study in April in English, French, German, Portuguese, and Chinese, which is a pretty good sampling of the languages spoken by the programmers of the world, but in all fairness, Spanish, Japanese, and several of the languages spoken in India are missing from the mix.

The resulting study, which you can read here, compiled enough data to be statistically significant for Adobe Creative Suite/Macromedia Studio, Codegear Delphi, Genuitec MyEclipse, IBM Rapid Application Developer, Microsoft Visual Studio, Oracle JDeveloper, and the Sun Microsystems sponsored, open source NetBeans as well as Sun Studio IDEs.

After paying $2.1 billion to acquire Rational in December 2002, you can bet that IBM is as pleased as punch to find out that its Rational Application Developer got the best scores from the users of this IDE who took the Evans Data survey, beating out JDeveloper, Visual Studio, and Sun Studio by a couple of points. “It’s safe to say that RAD’s users love their IDE and tools as well as the support and care they get from IBM,” the report states. “All of the IDEs in this survey are popular and very good tools, but RAD is just head and shoulders above them when it comes to user satisfaction.”

That seems to be a bit of a stretch if you look at the data in the report, which I suggest you do. Rational Application Developer had what looks like a composite score of 168, with JDeveloper, Visual Studio, and Sun Studio clustering around 165, Adobe Creative suite at around 163, Codegear Delphi at around 162, NetBeans around 160, and Genuitec MyEclipse at around 154. The report doesn’t say what the scale was for these, but this seems like pretty tight clustering as far as I am concerned. Across the 15 categories, developers who use Rational Application Developer ranked it with a high of near 200 points for the debugger to a low of around 140 on the integration of third party tools. What is clear from the charts in the report–and what is really useful for programmers who are looking at adopting a new IDE, or an IDE for the first time, which many are, is that IBM’s ratings across the 15 categories are more tightly clustered than others, raising its composite score. The MyEclipse tool, by comparison, rates almost as high as the Rational IDE in terms of its editor and debugger (pushing 200 points), but some features only rated at just north of 100 points, dragging the overall composite score down. This happened even more dramatically for the Delphi IDE, which rated very low for Web design tools and its profiler, but rated much higher than RAD when it came to ease of use, compiler performance, and the performance of the applications it creates.

JDevelopers’ bar chart looks almost the same as the one for RAD, and that explains why JDeveloper actually edged out RAD in the 2007 survey and the two were neck-and-neck in the 2006 survey. Visual Studio and Sun Studio are hard to distinguish from RAD and JDeveloper, but each one has slightly different data for each of the 15 categories.

While this comparison of IDE’s has some value, each IDE was rated by its own users and therefore not as a true comparison and contrast. If Evans Data wanted to do something interesting, it would have a team of programmers actually rate the IDE’s they don’t know in addition to the ones they do know. Actually pay people to make a comparison and shake out the true differences, as seen through the eyes of experts, between the different IDEs available. Just because one group of programmers are happy with their tools in certain ways does not mean they would not be happier using other tools if they even knew what made those tools different or better.

Still, in the absence of any better data–and Evans Data is one of the few places where you can get any sense of what is going on in terms of market share and programming practices–this annual IDE study will have to do for now.

This Issue Sponsored By

Table of Contents

Content archive

Recent Posts

Subscribe

Pages

Search