Hurricane Wilma Fails to Flush Roto-Rooter Down the Drain

Published: March 28, 2006

by Alex Woodie

"Roto-Rooter, that's the name, and away go troubles down the drain." That catchy jingle has been a life-saver for millions of Americans when plumbing problems strike, which they always seem to do, at the worst possible moments. But when Hurricane Wilma, a record-breaking category 5 storm, was threatening Roto-Rooter Service's South Florida call center last October, who would the company call to keep its vital OS/400 applications up and running, and its doors open for business? The answer might surprise you.

Over the past few years, Roto-Rooter Services, like many mid-to-large-size American companies, has engaged in a process of consolidating its operational assets and IT investments to cut costs and boost efficiency. For users of IBM iSeries servers like Roto-Rooter, the server side of this equation is a fairly well-defined and straightforward process, as Big Blue has consistently delivered new designs over the years that consistently pack more and more power into the same size boxes.

The latest round of server consolidation has kept Dave Stall, Roto-Rooter's hardware manager, busy since late summer 2005. Stall has traveled the country upgrading dozens of iSeries and AS/400 Model 170s located at the company's four primary locations to new i5 Model 520s. However, while server consolidation can save a company millions, the process can also expose new risks, in which Roto-Rooter was about to get a firsthand lesson .

And Then There Were Three

When Stall first started working at Roto-Rooter 13 years ago, the company was divided up into much smaller divisions, and each division had its own AS/400.

"We had 60-some AS/400s. You lose a machine, you lose a division, you lose one-sixtieth of your business," Stall says. No big deal, right? "When we started condensing call centers, we could do processing better and for less expense, but you get to the point where if you lose a call center, you lose one-third of your business." Now that's definitely a big deal.

Over the years, Roto-Rooter has consolidated 60 company-owned locations into three main call centers, which are located in Fort Lauderdale, Baltimore, and Chicago, as well as company headquarters in Cincinnati. Each call center employs 60 to 80 employees, who take calls from people all over the country and dispatch drivers out to fix their plumbing emergencies.

The call centers are not necessarily aligned geographically to the area they're in. For example, Roto-Rooter's Ft. Lauderdale office covers Tampa; Atlanta; Minneapolis- St. Paul; Cincinnati; Charlotte, North Carolina; and Norfolk, Virginia. Customers don't care that a problem occurring half a continent away could impact their ability to get their drains cleared, but that's the truth of the matter.

The Search for Higher Availability

Hurricanes aren't the only threat to South Florida, Stall says. For instance, lightning has struck and destroyed the electrical transformer outside the company's call center two or three times over the years, although the generator always kicked in, preventing downtime.

"Between the lightning and the hurricanes, we knew time was running out," Stall says. "We just felt there has been so many times that Florida had been just missed that we needed something in place."

Stall was left with a poor first impression of high availability products two or three years ago. The products seemed "cumbersome" to him, and his research cast doubt on whether the software could actually do the job when push came to shove. "I had read a lot of articles about companies that . . . had never done successful rollovers. They'd tried to do rollovers, but were never able to complete them. They always had problems," he says.

Stall had heard good things about a new class of high availability software based on IBM's remote journaling technology, so he decided to check out Echo2, a remote journaling-based high availability product from iTera. "We liked what we saw in the iTera product. It just seemed to be a lot more slimmed down, more menu driven, easier to go through checks and balances," he says.

It was late summer or early fall 2005, and Roto-Rooter had just made the decision to implement a business continuity plan that used Echo2 to replicate data from the Florida, Maryland, and Illinois offices to the headquarters in Ohio, where an i5 Model with sufficient processing headroom could handle the transactions of an affected site, or even two affected sites at the same time.

Trial by Hurricane

As part of the Echo2 implementation, Roto-Rooter sent Stall and other IT workers to Salt Lake City, where iTera experts remotely installed the Echo2 software on the company's OS/400 servers, configured the software, and trained the employees on its use. Then they returned to Cincinnati.

"We'd been back from class for two weeks," Stall says. "We were just getting used to the software, doing daily balancing, making sure all the records were updating properly. We got to the point where we knew where Hurricane Wilma was headed. So we asked iTera 'What should we do.'"

At this point, Echo2 was installed on Roto-Rooters' OS/400 servers, but the software was not considered live. Stall had yet to complete the most important test of all--the planned rollover. That was the final step in the training process, and a step that iTera's experts would walk him through. But with Hurricane Wilma headed directly for Florida, Stall didn't have the luxury of time.

At Stall's request, iTera stepped the process up a notch and helped the company prepare for its first rollover. The company had a couple of days to prepare, which they used to ensure key files were in place on the backup machine and everything was in synch before switching over.

On Sunday evening, October 23, 2005, Stall called iTera's tech support line and reached an iTera employee named Jan, who was at home in Utah cooking dinner. Jan walked Stall's team through the process of executing the rollover, which was completed successfully.

Stall was impressed with Jan's abilities. "She was so delightful to deal with," he says. "I laughed because she was teaching her husband how to make mashed potatoes, how to put salt in the water,doing this while helping us do a rollover. I couldn't think of any better service to get from somebody."

The Aftermath

On Monday morning, Hurricane Wilma slammed into South Florida as a powerful category three storm. With the abundance of hurricanes last year, perhaps people overlooked Wilma a bit, and she took a steep toll in the areas around Fort Lauderdale, Palm Beach, and Miami, including 35 lives.

The area around Ft. Lauderdale suffered severe damage. Electricity went out, streets were flooded, trees were uprooted, and street signals were knocked over. Suffice it to say, Roto-Rooter South Florida employees wouldn't be coming to work that day, or for the next week. However, because the company had transferred operations from the i5 in Ft. Lauderdale to the i5 in Cincinnati, Roto-Rooter's call center employees in Baltimore and Chicago were able to pick up the slack for the out-of-commission office, and there was no loss of business continuity. The call center employees never even knew they were logged onto the Ft. Lauderdale partition on the Cincinnati system.

The frame relay connection between Ft. Lauderdale and Cincinnati was down for almost 24 hours, and during that time, the Ft. Lauderdale partition recorded almost 1.6 million journal transactions. When the network was reconnected a day later, it would take about 45 minutes to re-sync from the Cincinnati box to the Ft. Lauderdale box, which was running on generator power.

The company wouldn't invite the bulk of its South Florida call center employees back to work for a week due to the extensive storm damage and the lack of reliable power (a plugged fuel filter on the generator, of all things, prevented the company from running the OS/400 server on generator power for a short period), but the server and the call center equipment would have been available almost immediately after the storm cleared had their been reliable power.

Stall received kudos from management for the way he and his team handled the storm. "It's always nice to have something work out as designed especially when we test it live," one senior vice president e-mailed to Roto-Rooter's CEO. "Everything worked as expected [although] we did not anticipate using it this soon."

In early March, Stall installed the final i5 Model 520 at the company's Chicago office, and by June, the company should have Echo2 running on all four systems. At that time, Stall plans to conduct a series of rollover tests to give his team practice for a disaster--just in time for another hurricane season.