Behind the Scenes at the Award-Winning iSeries Support Center

by Timothy Prickett Morgan

Last week, IBM announced that the iSeries Support Center, which provides technical support for the iSeries platform, competed for and won a prestigious award presented by the Service & Support Professionals Association (SSPA). The award that IBM's Rochester Lab was given was a great excuse for me to get an inside look at the iSeries Support Center and show you the people, applications, and processes that go into the technical support behind the OS/400 platform.

The Star Awards are a bit like the Academy Awards for technical support, and they are not restricted to IT companies. Any company that provides technical support and that is a member of the SSPA can get such an award. While the Academy of Motion Picture Arts and Sciences creates a long list of movies, acting and support categories when it presents its awards and then asks the members of the academy to vote on the movies and people it picked in those categories, the SSPA's Star Awards, which debuted in 1989, work a little differently. Rather than be nominated for a Star Award, you have to provide super-secret documentation (which is given under a non-disclosure agreement) that proves you are providing excellent technical support in given categories and then you have a Star Award bestowed on you by your peers. This is a much more rigorous process for picking the stars of tech support than the stars of the big screen. Imagine this: if the Academy Awards worked this way, an actor or actress would have to apply for an award, document specifically and scientifically why their performance was the best, and then be awarded the Best Actor or Best Actress by their acting peers who may also be competing for awards, not by a larger guild of actors, writers, directors, and producers. The SSPA currently has just under 200 member companies, and they are some of the biggest names in technology.

The SSPA doesn't go nuts with the categories like the Oscars does. At this year's awards, which were presented on October 11, IBM's iSeries Support Center was the winner of the Star Award for Sustained Performance, one of seven categories that included Most Improved (ProQuest Business Solutions), High Case Volume (Sage Software), Innovative Support (Oracle), Mission Critical Support (Dell's Enterprise Services unit), Complex Support Applications (RSA Security), WebStar Service Award (typically for complex applications, usually for engineering and scientific fields, also won by Oracle). The SSPA also gives out a Hall of Fame award every now and then, and this year, you guessed it, Oracle won that one, too. Oracle is the ninth hall of famer so far.

The OS/400 platform is, as we all know, legendary for a number of different things, the first being the highest customer satisfaction, year after year, in the server market. The other thing that the OS/400 platform is known for is reliability, stability, and ruggedness. While the development teams who create the hardware and software can take a lot of the credit for the reputation that the AS/400 has always had and the iSeries still has, that customer satisfaction rating has a lot to do with the iSeries tech support staff IBM has created to cope with things when they go wrong. Because as we all know, things not only can, but will go wrong, and when we least can afford them to, and when that happens, the difference between a rough day and a terrible one is going to be that tech support team on the other end of the telephone.

The Sustained Performance Star Award goes to the company that can demonstrate consistently high levels of tech support and customer satisfaction for three years. According to Jim Rubish, business executive for eServer support and manager of the iSeries Support Center in Rochester, Minn., to go for the Star Award for Sustained Performance, IBM submitted 50 pages of feeds and speeds describing its tech support operations from 2002 through 2004 and then threw in the year-to-date statistics for 2005 as well for good measure. To even be able to chase the award, Rubish had to submit a paper to the SSPA that outlined why IBM's iSeries Support Center should even be considered. Only a few companies that ask for consideration in each Star Award category are allowed to chase it. (And no one will divulge who IBM beat out in this year's competition.) Rubish says that the fact that IBM was being judged by its peers, who are sometimes IBM's competitors and who are not just a few members back at SSPA headquarters who run the organization, is important. "I think it really validates the results."

Rubish explains that the key metric that puts you over the top in technical support is customer satisfaction after a call is finished and the issue has been resolved. What IBM Rochester has done for a long time is randomly select people who have called in issues, and poll them on how their call went. This way, IBM has real-time data that tells it on a day-by-day basis not only what kinds of tech issues its customers and tech support staff are facing together, but also how well these situations are being resolved. There are many more metrics that go into managing a tech support operation, of course, such as the number of cases per month that are handled, how long each call takes, the percentage of cases that get fixed on the first call, and so forth.

These and many other statistics are tracked by IBM's own in-house developed technical support applications. IBM has hacked systems onto its own telephony systems that allow the initial support rep that takes a call to gather all the relevant information about the customer account once and then, as the call is routed around IBM's tech support systems, the customer information is automatically routed to the next technician in line, customers do not have to repeat all of that information, which shortens call length and keeps people from experiencing phone rage. The other thing that IBM Rochester does is assign a single point of ownership to each tech support problem as it is routed to the correct technician. This, says Rubish, is another key differentiator, since customers who are facing down a technical issue that could be disrupting their businesses do not want to be bounced around. Another key differentiator is the close proximity of the tech support staff to the development staff. This is not always a big factor, says Rubish, since the typical call placed by customers in the United States only takes one person to handle. But, the closeness of support and development people can be critical when a serious issue crops up.

iSeries support is different depending on where you are. First of all, if you are a direct IBM customer--which means you are one of the 15 percent of the OS/400 shops that are big enough to command Big Blue's attention--then when you make a support call, you go directly to the iSeries Support Center in Rochester, which Rubish says is called the "back-end" support center. (IBM doesn't use terms like Level 1, Level 2, and Level 3 support because it is not a call center that does simple things like helping people change passwords and such and because it has a flatter hierarchy than a lot of tech support organizations.) Customers who buy their machines from resellers and other business partners can get what IBM calls "front-end" support--meaning the place where you make the first call and someone tries to help you--from the support organizations at the resellers and partners, or, if the customer contract specifies it, IBM provides both the front-end and the back-end support. IBM doesn't charge extra for support for customers who chose IBM rather than partner front-end support when they buy from third parties; to its great credit, it is revenue and fee neutral on that issue.

Those who have front-end support from third parties obviously can have their issues quickly--and transparently--escalated back to the iSeries Support Center in Rochester, which has approximately 300 full-time staffers who take turns providing 24x7x365 live tech support coverage. Outside of the United States, partners or the tech support organizations within regions and countries often supply the front-end support, with the Rochester center providing back-end support. Because the overseas operations often involve tech support specialists who are certified to provide coverage on iSeries hardware and software as well as other IBM technologies, it is hard to give a precise number for the people who provide iSeries coverage, but it is in the hundreds.

At the back-end level, Rubish says that IBM fields about 1,000 calls a day. He says that this is a large enough call volume that IBM could have qualified for the high-volume award, but considering that there are over 200,000 OS/400 shops out there in the world and they are constantly changing hardware, operating systems, and other systems software, to my way of thinking this is not a very large number of calls relative to the size of the OS/400 installed base. And these are not necessarily easy calls--computers being the wonderfully useful nuisances that they are. But Rubish is, as you might expect from techies from Rochester, very calm, cool, and collected about it. "All of the clients we have are professionals, and there is no such thing as a dumb call," he says.

IBM has separate organizations for back-end support for various systems. The pSeries line has support centers in Dallas and Austin (which won a Star Award in the past for most-improved tech support), zSeries tech support comes out of Poughkeepsie, N.Y., and Linux support comes out of the former Sequent offices in Beaverton, Ore.. All customers have access to online technical resources (which they get for free), but it is the inside IBM systems that are what makes Big Blue's tech operations hum. These applications are all homegrown, including hacks to the telephony systems to track and route calls as well as the internal systems that tech support people use to field calls. The applications run on a variety of OS/400, AIX, and mainframe platforms, not just on the iSeries, because different IBM units in different geographies started out with different iron and skillsets, just like any big multinational corporation. Rubish says that from time to time, IBM looks at replacing its internal applications with third-party code, but he says that the level of functionality and fit with IBM's own tech support operations just isn't there.

So, the next time someone gives you guff about writing your own applications--including your IBM sales rep or business partner rep--you just remind them of this and then go right on writing your applications in RPG.

When I talked to Rubish, the thing I wanted to know is how do you get a skilled tech support technician? The answer is this: You have a lot of real experienced OS/400 gurus on site, and you train them. While IBM has a lot of deeply experienced midrange professionals in both the development and tech support facilities in Rochester, there is plenty of new blood there. In fact, Rubish says that IBM has created a rigorous training regimen to build out its tech support staff, like a kind of OS/400 Boot Camp--or, to be more precise, IPL Camp. IBM hires a bunch of compsci graduates and puts them through six to nine months of training in IPL Camp. After that intense experience, they can start fielding calls. But all the OS/400 expertise in the world is backing them up.