OSHA Changes To IBM Battery Handling Affect Cache Battery Replacement

September 19, 2012 Hey, Joe

I’m scheduling cache battery replacement for my production machine. Since this machine is never off-line, do you have any best practices or recommendations for how to change these batteries while production is running?

–Ben

Due to a recent United States Occupational Safety and Health Administration (OSHA) change, changing the cache batteries on your IBM i partitions isn’t as easy as it used to be, especially on a production machine. Here’s what happened.

IBM uses batteries in its disk controllers to provide disk drive caching. The batteries have a useful life of about 2.75 to 3 years and the system will start sending warning messages to change your batteries when the cache batteries are older than about 2.33 years. When one of these batteries dies, it may not cause a disk drive failure, but it will disable disk caching for the disk arrays the controller manages, and that will have a significant impact on hard drive performance.

The IBM procedure to change cache batteries is relatively easy, and it can be performed concurrently while your machine is running. This is valuable for machines like yours that may have very small maintenance windows and often can’t be taken down for hardware maintenance.

Before this year, IBM shipped its cache batteries fully charged. This assured administrators that there wouldn’t be any degradation with disk drive performance after the batteries were changed, because the batteries were fully charged on installation and ready to go. The IBM tech would flush the cache, replace the batteries, and your system would continue functioning as usual. So the cache batteries could theoretically be replaced any time the system was running, and IBM would routinely change these batteries on the fly on a running machine.

But due to recent OSHA requirements, IBM can no longer store and deliver fully charged cache batteries. IBM now has to store and deliver these batteries with little or no charge (the concern is that a warehouse full of charged batteries could explode). As a result, uncharged cache batteries are now being installed in IBM i machines. These batteries are then charged after installation, and it can take up to five hours for the batteries to reach their full charge.

The end result is that after cache battery replacement, disk drive caching is literally turned off on an IBM i machine until the batteries charge. This can result in temporarily slow disk drive access, which can delay critical processing, prevent orders from being fulfilled, and stop your organization from processing information in a timely manner. If you change batteries during a busy time, your users and customers may see a noticeable deterioration in response time, and information processing can significantly slow.

The good news is that after recharging (which again, takes up to five hours), the machine will suddenly snap back to life as disk caching resumes. I’ve experienced this scenario myself and processing time can be miserable while you’re waiting for the recharge to end.

Given this wrinkle, here are my new recommendations for changing cache batteries on an iSeries, System i, or Power i machine.

Batteries can generally be changed concurrently for development and backup machines that are not running production. A slowdown in disk processing on non-production systems may not be noticed quite as much while the batteries are charging. However if it’s critical that these machines be processing at their full speed, wait until a slower processing time or a maintenance window before changing the batteries.
For any production, development, or backup machine, don’t change the cache batteries immediately before running a full system backup, PTF application, or any other maintenance that requires a lot of disk reads and writes. This will have a significant effect on the completion time for these processes.
For production machines, don’t change the cache batteries during a busy period or during a batch processing window, when a lot of disk drive access is happening. This will significantly slow down throughput until the batteries finish charging.
If you can, change the batteries while the system is down or in restricted state. This is always the safest procedure for any kind of hardware maintenance, because you don’t run the risk of affecting processing if

The general point is that just because something can be changed concurrently, it doesn’t mean that it should be changed concurrently. If possible, plan your cache battery changes as thoroughly as you would any other hardware maintenance. Doing this can save your system lots of grief during busy processing time.

–Joe

Follow Me On My Blog, On Twitter, And On LinkedIn

Check out my blog at joehertvik.com, where I focus on computer administration and news (especially IBM i); vendor, marketing, and tech writing news and materials; and whatever else he come across.

You can also follow me on Twitter @JoeHertvik and on LinkedIn.

Joe Hertvik is the owner of Hertvik Business Services, a service company that provides written marketing content and presentation services for the computer industry, including white papers, case studies, and other marketing material. Email Joe for a free quote for any upcoming projects. He also runs a data center for two companies outside Chicago. Joe is a contributing editor for IT Jungle and has written the Admin Alert column since 2002.