Admin Alert: Common Mistakes When Failing Over to a CBU

August 13, 2008 Joe Hertvik

My shop ran its first high availability failover test in 2007. We’ve since run many more tests where we configured two different System i 550 Capacity BackUp (CBU) units to impersonate other systems. We’ve also made many mistakes in the process. Today, I’m focusing on common CBU failover problems and how to avoid them. Hopefully, this information will help make your failover process run smoothly.

Meet the CBU

In the i5/OS world, a CBU system is a specially configured Power i, System i, or iSeries machine that communicates with your production machine to continuously replicate production data and applications using high availability software. In case of disaster, the CBU can be switched over to “impersonate” the production box, servicing your users, devices, and companion servers with very little delay. When the main production machine comes back up, the CBU relinquishes its role and production is switched back to the regular system.

Because it’s delicate work to configure a real-time replacement machine that your entire organization may have to run on, most CBUs are exhaustively configured and tested to ensure that stupid mistakes don’t wreck your processing if the CBU ever has to pinch hit for the production server.

Enter Stupid Mistakes

There are three critical areas that either allow or prevent a CBU’s successful impersonation of a production i5/OS server:

Setting up your replication software to provide up-to-date copies of data and applications that reside on your production box.
Auditing the replication process to ensure that your CBU objects remain in sync with the objects on your production machine.
Creating and refining the run book that details the steps needed for your CBU to impersonate the production machine during a failover situation. Failover tests are needed to validate and improve the process that the CBU will use to impersonate production.

Given this broad outline, here are some simple mistakes that can blow your CBU implementation right out of the water. Avoid these issues and you’ll be sitting pretty. Trip them, and you’ll have a good-sized problem on your hands.

Replication and Auditing–Eyes On the Prize

When setting up replication, make sure that you’re replicating all the necessary objects your system relies upon, not just the data and programming libraries. These objects include:

All object libraries and fix libraries that run specific or critical pieces of your application software.
IFS folders and directories containing additional configuration and program files for your software. They may also contain stream file data in the AS/400 Integrated File System (AS/400 IFS) that your software needs to run.
DLO folders and objects for other PC-like files that your environment may need.
User profiles.
i5/OS devices that specially describe configured printers and other peripherals, complete with IP address, identifying parameters, and other custom configurations.

The first common mistake is neglecting to audit your object replication. If your objects get out of sync, you will lose necessary data or program objects when you failover. So take full advantage of your replication software’s auditing functions, and check your audit reports every day. There is no other way to ensure that your objects remain in sync. When auditing, beware of out-of-sync objects and production system libraries that are not present on the CBU partition.

New libraries are especially critical to replicate, as applications programmers frequently add new functionality that must be ported over and set up on the CBU. On a regular basis, you may want to cross-reference the list of libraries and IFS directories that you are replicating to the CBU with the list of all the libraries and IFS directories on your production box. This comparison will help you catch any new libraries or directories that need to be added to your replication list.

Over-Replicating

While auditing helps ensure that you have a complete replicated copy of your database and application software, it is possible to replicate too many objects, particularly for third-party software packages. Be careful to exclude these items from your replication scheme, or they will cause a problem when you failover.

License keys for third-party software. i5/OS license keys are usually specific to the serial number of the machine that they are running on. If you’ve installed a second copy of a package on the CBU, complete with its own license key, you don’t want to overwrite it and disable the software by replicating a different license key from the production partition. Check with vendors to determine which program files and objects (including the key file) are machine specific and should not be replicated.
System libraries and IFS folders that have nothing to do with your application programs or data. In particular, be very careful about replicating objects in any library whose name begins with the letter ‘Q’. If you need to replicate objects in the general purpose (QGPL) or IBM user libraries (QUSRSYS), be very specific as to what you include or exclude from your replication, as you may inadvertently damage your CBU system. On the AS/400 IFS, watch out for overwriting files in the /QIBM folder.
Third-party vendor packages that shouldn’t be replicated because you are already running a different instance on your CBU. The best example here is Help/Systems’ Robot/SCHEDULE software, which is used to sequence and run automated jobs on an i5/OS system. When failed-over to the CBU, you will need to run the SCHEDULE instance from your production box (complete with all your production entries). However, when the CBU isn’t failed-over, it may need to run its own instance with schedule entries for backup jobs and other CBU housekeeping. In cases where you run two different instances of a package on your production and CBU partitions, you need to contact your software provider and work out instructions for reinstalling the production instance on your CBU during failover.

Run Book Issues

As you put together the run book (your bible for failing over), you may wind up working with a consultant who will provide you with a basic run book for failing over. This book will contain all the information you need to failover to the CBU, have your CBU impersonate your production machine, and then fail back to the production box.

However, you’re going to need more than the basic run book to fully failover to the CBU. Once the initial run book is finished, you’re going to have to put in your own run book instructions to set up the CBU exactly the way it must look to successfully impersonate the production box. Some of the additional items you may need to add to the run book include:

How to change communication parameters so that remote devices and partitions can talk to your CBU. This is particularly important if you’re developing on a separate partition and promoting those changes to production through a change management program, such as Aldon’s Lifecycle Manager. You may also encounter this situation with some i5/OS fax serving products that require you to change an Ethernet address inside their configurations to match the Ethernet card you’re running on the CBU.
Replacing the i5/OS System Distribution Directory (SDD) on your target machine with the SDD from your production machine. The SDD plays an important role in distributing output from many i5/OS products, and some products will not work for your users without having appropriate SDD entries in place.
Changing the i5/OS NetServer server name (formerly AS/400 NetServer) on the CBU to reflect the NetServer name of the production box. Many applications use NetServer to access i5/OS data.
Reinstalling/reconfiguring third-party software instances of popular programs that run on both the CBU and on your production box. As noted above, if you’re running a program that has custom parameters for both your production and your CBU partitions, you’ll have to include run book instructions for reconfiguring it on your CBU. The Help/Systems’ Robot/SCHEDULE example cited above falls into this category.
Retrieve any necessary temporary software keys from third party vendors. For some packages, you don’t need to purchase a second production license or a disaster recovery/high availability license to run the product on your CBU during failover. Instead, the package may run for a few days on a grace period, and you may call the vendor if you need to run the product for longer than the grace period. You should include specific instructions for how to handle this type of licensing in the run book.
Obtain temporary keys for certain pieces of IBM software. Because of CBU licensing, IBM may not provide keys for running certain i5/OS products on the CBU. If you want to use these products in a failover situation, you may need to call IBM’s Key Center to get a temporary key to run your product. Common IBM products falling into this category include Query (5722QU1) and DB2 Query Manager and SQL Dev Kit (5722ST1).

There’s a lot more to failing over than just changing your CBU’s identity to impersonate the production box. Hopefully these tips will help you understand and avoid some key mistakes that can prevent your failover process from running successfully.