Admin Alert: Dealing with i5 Critical Storage Errors,

May 2, 2007 Joe Hertvik

Part 2

In the last issue, I discussed some tricks for monitoring your system to detect storage problems. Today, I want to shift the focus from determining when your disk storage is filling up to what you can do to increase available drive space. I’ll do this by providing a checklist of several different techniques you can perform to retrieve system storage space that you might have thought was lost forever.

Finding the Right Cure for Your Storage Woes

Disk drive capacity issues can cause system problems when i5 storage usage passes the 90 percent level. Performance degradation starts to occur when your drives are 90 percent full, and when storage usage passes 95 percent, the system can crash. This is why i5 system administrators should monitor for critical storage situations and take action before their hard drives fill up and endanger the system. When dealing with storage errors, the prudent administrator should understand and use the following tools and techniques.

i5/OS Storage Monitoring System Values and Settings, which can be configured to alert you when storage usage and availability pass specific threshold values.
Know how to spot storage problems caused by disk fragmentation and storage directory errors. Storage problems sometimes occur when the i5 system directory contains too many entries or disk drive fragmentation reaches what IBM calls “a point of significant concern.” These problems usually show up when the system receives a CPI0999 error. I outlined several techniques for handling this situation in a previous article called The Case of the Mysterious CPI0999 Error. All of these techniques deal with handling storage issues on a system-wide basis for all hard drive objects. These options include reclaiming damaged and incomplete objects, reorganizing your disk drives, and IPLing your system.
Become familiar with how to reduce storage usage by modifying various system objects that are consuming more storage than they should. The rest of this article concerns itself with different ideas on how to reclaim system storage by reviewing commonly found objects and modifying their system storage footprint.

Everyday Ways to Decrease Storage Usage

If you find that your system storage is approaching 90 percent usage, here’s my checklist of quick ways to decrease storage usage. Depending on how your system is configured, you might be surprised at just how much storage comes free when you examine these items.

1. Delete excessive spooled files: Are there output queues such as QEZJOBLOG that contain too many spooled files? From the green-screen, you can look at the total number of files in each output queue by running the Work with Output Queue command (WRKOUTQ) like this:

WRKOUTQ OUTQ(*ALL)

If you find that there are certain output queues that contain an excessive number of spooled files, you can clear out all the items in a specific output queue by using the Clear Output Queue command, like this:

CLROUTQ OUTQ(library/output queue name)

Or you could use iSeries Navigator (OpsNav) to sort and delete spooled files in multiple output queues according to the number of pages in each spooled file, spooled file status, user specific data, etc. If you are more ambitious, you can even set up an automated routine that will selectively delete spooled files according to your own specific criteria.

Also please be aware that there is a specific i5/OS condition that has been known to produce thousands of joblogs without any known explanation. This condition (which I have nicknamed Joglogapalooza) can usually be identified when hundreds of CPF22E3 messages start appearing in your QSYSOPR message queue. A more complete description of this problem and how to handle it can be found in an article that I wrote called Joblogapalooza, Its Possible Causes and a Call for Input.

2. Check and delete unnecessary journal receivers: If you’re doing a lot of journaling in a particular library, your journal receivers can fill up and become fairly large. Review the journal receivers in your library and check to see whether any of the receivers are getting too large. Then if your company allows you to delete journal receivers (some companies have strict standards on how long to keep receivers due to laws like Sarbanes-Oxley), you can detach the filled journal receiver from your journal object, replace it with another newly generated receiver, and then delete the original journal receiver to retrieve some storage space. To detach a current receiver and replace it with another receiver, you would run the following Change Journal command (CHGJRN).

CHGJRN JRN(library/journal_name) JRNRCV(*GEN)

The Journal Receiver parameter (JRNRCV) tells i5/OS to automatically generate a new journal receiver and attach it to your target journal. Once the old receiver is generated, you can delete the old receiver if company standards allow it.

3. Delete junk libraries and files: Some i5 administrators (myself included) have a bad habit of backing up entire libraries to disk (rather than to tape media) for quick restores when they are making a large number of changes to that library. Similarly, they may also back up large files to disk when they are changing a lot of records in that file. While these duplicated objects can be handy in the short run, the problem is that they will clutter up your hard drives in the long run and should be eliminated as soon as possible after usage. You may also have some other libraries that were used for installing software or exchanging information with another system. These libraries may also be good candidates for deletion. So another way to reclaim hard drive space is to go on an i5 scavenger hunt for any unnecessary libraries and duplicate files that can be deleted. Once you find a library that can be deleted, make sure that you have a good backup of it to tape or other media and then run the following Delete Library command (DLTLIB).

DLTLIB LIB(library_name)

4. Clear save files that you don’t need: Sometimes administrators will back up entire libraries to a save file in order to FTP a library between i5 partitions. Alternatively, software vendors may deliver installation files or maintenance release changes (MRs) inside save files that you FTP to your system. Check for any junk save files like these that can be cleared or deleted. You can find a list of all save files on your system by running the following Work with Files command (WRKF).

WRKF FILE(*ALL/*ALL) FILEATR(SAVF)

Once you have a list of all your files, you can check the size of any suspect save files and clear or delete those save files at will. To clear a save file, you would use the following Clear Save File command (CLRSAVF).

CLRSAVF FILE(Library/Save_file_name)

5. Other options for reclaiming space: You can also reclaim hard drive space by performing functions such as deleting performance data, reorganizing large physical files with lots of deleted records, and setting up your physical files to reuse deleted records when a new record is inserted into that file. All of these techniques are explained in another Admin Alert article called 5 Things to Do Before Adding iSeries Disk Capacity.

The techniques I reviewed can make it easier to control hard drive usage as it approaches its maximum storage capacity. In most cases, you should be able to reclaim some hard drive space and hopefully delay the time when you have to take the final step to solve disk storage problems: buying more disk drives for your partition.

About Our Testing Environment

All configurations described in this article were tested on an i5 550 box running i5/OS V5R3. Most of the commands used here are also available in earlier versions of the i5/OS and OS/400 operating systems, so the configurations should be usable in prior releases. However, you may notice minor variations in pre-V5R3 copies of these commands. These differences may be due to incremental command improvements that have occurred from release to release.