You Have Life Jackets, But Have You Ever Put One On?

The rain continued to fall heavily for the fifth consecutive day. I woke up the morning of June 28, and since I wasn’t feeling well, I called in sick. I told my boss to give me a shout on my cell phone if there were any problems at work. I remember dozing off on my couch with the Weather Channel chirping in the background. A few hours later, I heard the beeping of the emergency broadcasting system on the television indicating that severe flooding was taking place in most parts of upstate New York.

At first I didn’t believe it. This cannot happen to us, I thought. I jumped into my car and drove down to the plant. There was six inches of water in the parking lot in the morning and the water continued to rise. I ran through the water and met with the operations manager in our computer room. I told him to make sure to get the latest backup tapes for our AS/400 and other servers and get out of the plant before everything was destroyed. By morning, the plant had taken in six feet of water. The area that once represented our data center now resembled a swimming pool.

I started working at Amphenol in July 2005 as a Web developer. I had dabbled for about five years with Web development and decided I needed a job change. My role with my previous employer was a senior systems administrator for the AS/400. I had tons of experience–19 years to be exact–and I knew what challenges were ahead to try and get our AS/400, which was home to most of our critical business data, up and running again. I packed my car full of AS/400 tapes and headed off to our disaster and recovery site down in New Jersey. As I drove, I started to formulate my plan.

Hopefully, this will never ever happen to you. But what if it does? Would you be ready? Would you know what to do? This article addresses areas that I feel you should be familiar and comfortable with before something like this blind sides you. They are backup strategy, software keys, disaster planning, and navigating around the box.

Backup Strategy

I know many of you are saying, Big deal, we back up our system. That’s great, but do you really know what is being backed up, how often, and why? In the ideal world, it would be great if we could do a full system backup on a daily basis. Unfortunately, most companies run 24 hours a day, 7 days a week, and upper managers will start to grimace when they hear that the backup will take more than a few hours. Take the time to sit down and review how you back up your business data.

A few major points I’ll make regarding backup are:

• Understand SAVSYS: this backs up Licensed Internal Code (LIC), OS400 or i5/OS (the AS/400, iSeries, or i5 operating system), user profiles, private authorities and configuration objects. Without this tape, you cannot bring online your existing AS/400 at the hot site. This backup doesn’t have to be done on a daily basis, but should be done at least once a month.
• Understand SAVLIB and SAVCHGOBJ: Most backup procedures from companies I know of will backup all of the data libraries on a weekend using SAVLIB, and use SAVCHGOBJ to save the changes during the week. You need to examine how your company conducts its library saves, because that process will impact how things get restored.
• Logical files: Be aware that physical files need to be restored before logical files.
• Get familiar with the IBM Backup and Recovery Guide: This book goes into detail on saving and restoring your system.
• Backup software: If you use software to help with your backup and recovery strategy, get a solid understanding of how tapes are labeled, how things get backed up and how to read the logs.

Software Keys

You’ve got your AS/400 back up and running, and you are thinking that wasn’t so bad. Now, you’re going to tell users to go ahead and get back to work. The problem is, you might encounter a message saying to enter a new software license key to use the software.

Long ago, you could save a software library and restore it on another AS/400. Times have changed. The majority of the software I see today is based on some combination of the serial number, processor feature, and model number of the machine. The activation code that the software vendor gave you will not work if the serial number and model have changed. It is the vendor’s way of helping cut down on unauthorized copies of its software floating around.

To help in the future, make a list of all of the non-IBM AS/400 software vendors that you use, a contact person, your customer number, and a support phone number to call. Have the new AS/400 serial number, processor feature, and model number handy. These are system values QSRLNBR, QPRCFEAT, and QMODEL.

Disaster Planning

It is important to convince the management group in your organization that disaster and recovery is critical from the IT side of the house. People don’t realize how important email, phones, critical business systems, and networks are until you walk into your office and none of them work.

There are several good disaster and recovery sites. You would have to call one for specifics, but most allow you to once a year try and restore your systems at their facilities. It would give you an opportunity to “test” the steps to restore your AS/400, will expose things that you might not have thought about in your planning process, and give you valuable experience that might prove worthwhile down the road. Yes, it costs money, but consider the cost of a disaster if you aren’t prepared.

Navigating Around the Box

I’ve worked my way up the chain from tape ape to project management in the AS/400 arena. These are some tidbits I’d like to share with you that might help you in the event you have to recover from a major disaster:

• Plan on the restore of your data to take approximately one and a half times longer than the save backup.
• Print out a rack configuration for future use. I used it primarily for resource names and card types that were on the old system.
• Always have an external e-mail account. We lost our mail server for a month and I needed a way to communicate with vendors and the company.
• What types of modems or external devices do you have connected to your AS/400? Document what type they are and how they are configured.
• Make sure you have a list of contact phone numbers for your organization.
• Backup and Recovery Guide–be familiar with it. When following the steps to restore your system, you must follow the directions exactly.
• Have some basic knowledge of the CFGTCP (Configure TCP/IP) command. Some basic knowledge about IP addresses, gateways and host tables will help when you are trying to get servers and printers back up and networked into your AS/400.
• Get the fax number, if available, of your closest disaster and recovery site.
• Document everything for future reference.

I had the opportunity to work with two live disaster situations as a consultant before this one. The first time I was helping a company down in Long Island whose business was destroyed by a fire. I was in shock at how much work was involved. I managed to get through it after a week. The second time I had helped a Florida insurance company, that was destroyed by a hurricane, restore an AS/400 with 11 partitions in 5 days and it was much easier because I was prepared.

If you keep some of these things in mind, you will be ahead of the curve. The first thing to do is convince management why planning for a disaster, and the recovery from it, is critical to the success of your business. Explain how backups are important and taking a few hours of time during the day could be a life saver. Without a backup, you have nothing to restore. Think about software keys and vendor software support–do they have support on weekends, or is it just 8 to 5, Monday through Friday? How will not being able to use the vendor software impact your business?

The last thing is experience. I am a lucky person because my close friend and mentor, Jeanette McCabe, gave me opportunity each day at work to learn. Many of the concepts that I fumbled with years ago helped me get my company up and running in three days. I owe her a great deal for getting me where I am today. My goal was to learn something new each day, regardless of how minor it might have seemed at the time. My success with this recovery from disaster wasn’t luck. I succeeded because I was prepared and tried the life jacket on before getting aboard.

Bob Butcher is a software consultant with over 19 years experience in the AS/400 world. He specializes in client/server applications that deal with the AS/400 and Active Server Pages or Visual Basic. He can be reached at bbutcher@samicsoft.com.