Which Job Is Filling Up My System Storage?
August 21, 2013 Hey, Joe
Every so often, our system storage breaches its threshold level, and we have to find the job that is filling up storage before it crashes our system. This is usually a race against time. Do you have any tips on how to quickly find runaway jobs that are gobbling up storage?
I feel your pain on this one. Hitting the threshold can be a recurring event in some shops, and it’s always better to find out sooner when this problem occurs rather than later.
Here’s how I handle this issue.
Set Your ASP Threshold At A Reasonable Level
On a newly installed IBM i partition, the default ASP storage threshold is set at 90 percent. When your storage usage goes over 90 percent, a message is sent to the QSYSOPR message queue indicating that the ASP storage pool is almost full. But by then it can be too late, as the system will get more unstable as storage rises above 90 percent. When the system runs out of available disk for processing, your partition will automatically shut down.
So the first step is to check your ASP storage threshold and see if it’s at a comfortable level to give you enough warning if storage starts filling up unexpectedly. On IBM i 6.1, you can set the ASP storage threshold either through the system’s Dedicated Service Tools (DST) or the System Service Tools (SST). Follow these steps to get to and modify your ASP storage threshold in SST.
This process will change the threshold value for when the system should send an alert to the system operator message queue (QSYSOPR). Lowering it will give you more time to look for run-away jobs that are filling up storage.
Enlist Your System Monitoring Software To Help
Most system monitoring packages have special alerts that can send email or text messages when an unusual system storage issue occurs. In the monitoring package that I use, I can set up the following disk storage alerts to notify my staff when these error messages occur:
By doing this, my monitoring software helps me determine when the system is misbehaving and may be ready to overflow its disk drives. It gives me a better chance to find a growing issue before it becomes a system threatening problem.
So be sure to check your own system monitoring software for similar capabilities and activate disk monitoring to get an early wanting when problems develop, if you can.
Finding The Rogue Job
Once you get a message that system storage usage is growing at an alarming pace, it’s time to start hunting for the out of control job. Many times, it can be a user query, a looping job that’s generating too many spooled files or job logs, or an output file that’s getting too big.
The best tool for hunting for out of control jobs is the old fashioned green-screen Work with Active Jobs (WRKACTJOB) command. WRKACTJOB shows all the jobs running in your system and its sort capacity will help you find jobs that may be misbehaving. Here’s my drill for using WRKACTJOB to locate and end jobs that may be running wild.
These techniques should help you get an early warning when your disk drives fill up, and then identify the jobs that are causing problems.
Follow Joe Hertvik on His Blog, on Twitter, and on LinkedIn
Check out Joe’s blog at joehertvik.com, where he focuses on computer administration and news (especially IBM i); vendor, marketing, and tech writing news and materials; and whatever else he come across.
Joe Hertvik is the owner of Hertvik Business Services, a service company that provides written marketing content and presentation services for the computer industry, including white papers, case studies, and other marketing material. Email Joe for a free quote for any upcoming projects. He also runs a data center for two companies outside Chicago. Joe is a contributing editor for IT Jungle and has written the Admin Alert column since 2002.