fhg
Volume 8, Number 20 -- May 21, 2008

Admin Alert: Monitoring the Monitors

Published: May 21, 2008

by Joe Hertvik

Recently, my shop had to create a simple monitoring program to ensure that a critical subsystem was always running. This week, I'll review the process that we used to create that program and I'll also show you how to perform basic monitoring by using just a few simple CL commands. With some modification, these concepts can be applied to any situation where you can't use an off-the-shelf monitoring program.

The Problem I Solved

The system problem this solution addresses reminds me of the old saying about who watches the watchmen. On one of our machines, we use a third-party monitoring, paging, and email package to send out emails whenever a system problem occurs. However, due to some configuration problems after our backups, the machine didn't always restart the package and we wanted to set up a second email monitor to ensure that our monitoring software was always up. After all, if your monitoring software isn't up, you can't exactly use it to send out email alerts that it's down, can you? In this case, we needed something to watch the watchman.

Our shop already had SMTP configured on the target partition, so it wasn't a big issue to actually send the email with the proper commands or APIs, which is a simple way of saying that you'll need an active SMTP server and the i5/OS Mail Server Framework (QMSF) to run this program. The programming itself is a matter of using the right tools to monitor the situation and to send out an email, when appropriate.

To check whether the package was running, we wanted to check whether the subsystem that ran the package was up and running. This is relatively easy to do by using the Work with Subsystem Jobs (WRKSBSJOB) command, like this:

WRKSBSJOB SBS(subsystem_name)

If the subsystem is up and running, WRKSBSJOB displays all the jobs in the subsystem, similar to what you would see when you run the Work with Active Jobs command (WRKACTJOB) with the subsystem parameter.

WRKACTJOB SBS(subsystem_name)

However, the big advantage that WRKSBSJOB has over WRKACTJOB is that when you're using it to check subsystem jobs, it displays the following CPF1003 error if the subsystem isn't active.

CPF1003 - Subsystem &1 not active

Because the CPF1003 error is generated when you're checking the status of a subsystem that isn't active, that makes the WRKSBSJOB a perfect candidate for my simple monitor package. In my utility, I use WRKSBSJOB as the element that monitors for a running subsystem and sends an email out when the subsystem is not active.

Anatomy of a Monitoring Program

The name of the monitoring program is MONSBS, and it contains the following code.

PGM 
DCL        VAR(&SYSTEM) TYPE(*CHAR) LEN(8) VALUE(*BLANKS)
DCL        VAR(&SUBJECT) TYPE(*CHAR) LEN(44) +
             VALUE(*BLANKS)
DCL        VAR(&SUBJECT1) TYPE(*CHAR) LEN(35) +
             VALUE(' :SBS subsystem_name NOT +
             ACTIVE')

WRKSBSJOB  SBS(subsystem_name)
MONMSG     MSGID(CPF1003) EXEC(DO)
RTVNETA    SYSNAME(&SYSTEM)
CHGVAR     VAR(&SUBJECT) VALUE(&SYSTEM *CAT &SUBJECT1)
SNDDST     TYPE(*LMSG) +
             TOINTNET(('email_address@domain.com) +
             DSTD(&SUBJECT) +
             LONGMSG('PLEASE  RESTART THE SUBSYSTEM')
             ENDDO

DLTSPLF    FILE(QPDSPSBJ)
MONMSG     MSGID(CPF0000)

ENDPGM

To make MONSBS work, each piece of code performs the following functions.

  • The three DECLARE statements (DCL) create program variables that will be used in the email I'll be sending out when the subsystem isn't up and running.
  • The Work with Subsystem Jobs (WRKSBSJOB) command checks to see whether there is a subsystem named subsystem_name up and running on the system. If subsystem jobs are present (running), the program doesn't do anything else and it ends.
  • Using the Monitor Message (MONMSG) command, the program monitors for message CPF1003, which is issued when you try to use WRKSBSJOB to display jobs in a subsystem that isn't active. When CPF1003 is detected, the code drops into a DO group command that uses other commands to build an email message that will be sent to the target user.
  • The Retrieve Network Attributes (RTVNETA) command retrieves the system name and saves it in the &SYSTEM variable. I'm retrieving the system name for my email because I have several Power i/System i systems running at the same time, and I want to differentiate which system the message is coming from.
  • The Change Variable (CHGVAR) command builds the subject line of the email in the &SUBJECT variable. It does this by concatenating the system name (as retrieved in the &SYSTEM variable) with the ':SBS subsystem_name not active' literal in the &SUBJECT1 variable, to make the following subject line.
System_name: SBS subsystem_name not active'

Once I have my subject line created, I can send the email. There are many different ways to send an email on an i5/OS system. For this program, I chose to use the Send Distribution (SNDDST) command, which works with my SNADS configuration. To use SNDDST to send an email, I coded the following parameters:

  • The Message Type (TYPE) parameter is *LMSG (Long Message), which supports sending a distribution to an email address. This is mandatory for this program.
  • The Internet Recipient (TOINTNET) parameter contains the email address of the user that I want to send my subsystem down message to. If I wanted to, I could enter a list of addresses to receive the distribution. In addition, each address contains a companion value called Recipient Type that specifies whether the message should be sent with the recipient being designated as the primary recipient (*PRI), a copied recipient (*CC), or as a blind copied recipient (*BCC).
  • The Description (DSTD) parameter will be used as the subject line of the email. This parameter uses the &SUBJECT variable as its value, and the value of &SUBJECT will show up in the email subject line. Note that the DSTD parameter is limited to 44 characters, so the &SUBJECT variable must be shorter than 44 characters.
  • Since we are sending this email distribution as a long message type (*LMSG), the body of the email must be contained in the Long Message (LONGMSG) parameter, not in the Message (MSG) parameter. So I put the body of the email message into this parameter.

Once the SNDDST completes, the email message informing my recipient that the target subsystem is down has been sent. In order to use SNDDST, the user running this program must be enrolled in the System Distribution Directory (SDD).

As I noted, you can use any other i5/OS technique to send emails, and you are not limited to using the SNDDST command for this program. You could use the commands that come with many third-party email packages, such as RJS Software Systems' SMTP/400 product or Gumbo Software's Gumbo Mail package. You can also send email by using the i5/OS Send Mime Mail API (QtmmSendMail) or the JavaMail components that are shipped as part of the IBM Developer Kit for Java (i5/OS V5R2 or above). There are a number of different alternatives for sending your email. You can read about and find links for all of these techniques in the IBM eServer Email manual.

The end of the program performs some house cleaning to delete a QPDSPSBJ spooled file that might be generated when running this program in batch mode. To provide constant monitoring for my target subsystem, I run this program out of Help/Systems' Robot/SCHEDULE package every half-hour. If you don't have Robot/SCHEDULE or another job scheduler on your system, you can also use the native i5/OS Work with Job Schedule Entries (WRKJOBSCDE) command to add a job scheduler entry to enable this job to run at certain times during the day.

MONSBS is a nice little skeleton program that easily be modified to produce emails for a number of situations.


RELATED RESOURCES

Gumbo Software Mail Product Page

Help/Systems' Robot/SCHEDULE Product Page

IBM eServer Email manual

RJS Software Systems' SMTP/400 Product Page



                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot


Sponsored By
GUILD COMPANIES

Internet Programming for AS/400, iSeries & System i

Available NOW from the IT Jungle Bookstore

This guide from author Hideyuki Yahagi, an IBM Certified IT Specialist
with Internet and open source programming expertise, is suited for
programmers with traditional skills who want to quickly learn to use
the built-in Web serving capabilities of the System i.

Progressing from basic to advanced, this tutorial includes
programming tips, snippets of sample code, and a CD.

Price: $49.95
Buy Now!


Senior Technical Editor: Ted Holt
Technical Editor: Joe Hertvik
Contributing Technical Editors: Edwin Earley, Brian Kelly, Michael Sansoterra
Publisher and Advertising Director: Jenny Thomas
Advertising Sales Representative: Kim Reed
Contact the Editors: To contact anyone on the IT Jungle Team
Go to our contacts page and send us a message.

Sponsored Links

Aldon:  Keep your organization in line with Application Lifecycle Management solutions
COMMON:  Join us at the Focus 2008 workshop conference, October 5 - 8, in San Francisco, California
Vision Solutions:  System i Management Tips Blog - Free i5/OS Tips Each Week!


 

IT Jungle Store Top Book Picks

Easy Steps to Internet Programming for AS/400, iSeries, and System i: List Price, $49.95
Getting Started with PHP for i5/OS: List Price, $59.95
The System i RPG & RPG IV Tutorial and Lab Exercises: List Price, $59.95
The System i Pocket RPG & RPG IV Guide: List Price, $69.95
The iSeries Pocket Database Guide: List Price, $59.00
The iSeries Pocket Developers' Guide: List Price, $59.00
The iSeries Pocket SQL Guide: List Price, $59.00
The iSeries Pocket Query Guide: List Price, $49.00
The iSeries Pocket WebFacing Primer: List Price, $39.00
Migrating to WebSphere Express for iSeries: List Price, $49.00
iSeries Express Web Implementer's Guide: List Price, $59.00
Getting Started with WebSphere Development Studio for iSeries: List Price, $79.95
Getting Started With WebSphere Development Studio Client for iSeries: List Price, $89.00
Getting Started with WebSphere Express for iSeries: List Price, $49.00
WebFacing Application Design and Development Guide: List Price, $55.00
Can the AS/400 Survive IBM?: List Price, $49.00
The All-Everything Machine: List Price, $29.95
Chip Wars: List Price, $29.95


 
The Four Hundred
The Demographics of i Sales and Shipments

The i Edition of the BladeCenter S Finally Launches

HP More Than Doubles Services Biz with EDS Acquisition

Mad Dog 21/21: Saying No No No

A Word Cloud of IBM Server Brand Names

The Linux Beacon
NYSE Euronext Trades Mainframes and Unix for Linux and X64

Canonical Founder Calls for Synchronized Linux Releases

AMD Ships Low-Power Barcelonas as Two More Execs Exit

New and Updated Barcelona Boxes Debut from Sun

VMware Tweaks Virtualization Stack, Boasts of Greenness and Sales

Four Hundred Stuff
Symantec Combats Phishing with New Services Offering

BCD Slings a New C#-Based GUI with Catapult 7.0

SkyView and Innovatum Formalize Partnership with New Product

Profound Eliminates OLTP Requirement with Web Enablement Software

140 Apps and (Hopefully) Counting for i 6.1

Big Iron
NYSE Euronext Trades Mainframes and Unix for Linux and X64

Top Mainframe Stories From Around the Web

Chats, Webinars, Seminars, Shows, and Other Happenings

System i PTF Guide
May 17, 2008: Volume 10, Number 20

May 10, 2008: Volume 10, Number 19

May 3, 2008: Volume 10, Number 18

April 26, 2008: Volume 10, Number 17

April 19, 2008: Volume 10, Number 16

April 12, 2008: Volume 10, Number 15

The Windows Observer
Microsoft Patches Zero Day Flaw in Windows

HP More Than Doubles Services Biz with EDS Acquisition

Massive Expansion in Progress at Microsoft Data Centers

Microsoft Gives Customers a Break on New SMB Windows Packages

AMD Revises Opteron Roadmaps, Pushes Out Rev Gs

The Unix Guardian
New and Updated Barcelona Boxes Debut from Sun

HP More Than Doubles Services Biz with EDS Acquisition

Java Performance Is OS Agnostic on Power6 Gear

As I See It: Soothing the Savage Programmer

VMware Tweaks Virtualization Stack, Boasts of Greenness and Sales

Four Hundred Monitor
Four Hundred Monitor's
Full iSeries Events Calendar

THIS ISSUE SPONSORED BY:

WorksRight Software
Help/Systems
Guild Companies


Printer Friendly Version


TABLE OF CONTENTS
Use PCOMM Scripts to Dynamically Build a Spreadsheet, Part 2

Use SQL to Strip Out Tab Characters

Admin Alert: Monitoring the Monitors

Four Hundred Guru

BACK ISSUES

From the IT Jungle Forums
SNMP Traps on i5OS

Java Messages

Copying recs from a subfile to a file and keeping highlights

Imbedded SQL

CPYFRMSTMF problem

Restrict *cmd to specific user





 
Subscription Information:
You can unsubscribe, change your email address, or sign up for any of IT Jungle's free e-newsletters through our Web site at http://www.itjungle.com/sub/subscribe.html.

Copyright © 1996-2008 Guild Companies, Inc. All Rights Reserved.
Guild Companies, Inc., 50 Park Terrace East, Suite 8F, New York, NY 10034

Privacy Statement