Admin Alert: Combating Cross-Server Failures for the i5 Manager

January 10, 2007 Joe Hertvik

System i5 software doesn’t exist in a vacuum. It cooperates and cohabitates with other servers, triggering cross-platform processes that kick off alerts, update files and Web sites, and move information across the enterprise. Unfortunately, i5/OS inter-server communication can sometimes break down in unexpected ways. This week, I will provide a checklist of common communication breakdowns that can affect how your System i5 machine functions in a cross-server world.

There are three common scenarios for cross-server processing between a System i5, iSeries, or an older AS/400 and a companion server–such as a Windows or Linux machine or even an application running on another i5/OS partition:

The companion server provides the presentation and application functions, while the database is hosted by and accessed from your i5 partition. Database requests frequently originate from Web servers, database servers, email servers, and FTP clients. Data can also be requested from machines that provide services for your outside business partners, such as vendors and sales people. In this case, the application has moved off your System i5 to its companion server and the i5 box functions as a database server.
Your i5 box provides the presentation and application functions, while the database is hosted by and accessed from a companion server. This is the opposite of the situation in point one.
Your i5 core business processes or a companion server needs to communicate with another i5 process running on the same or another i5/OS partition, in order to retrieve information. In this scenario, your i5 partitions may function as both the application and database servers.

In all scenarios, cross-server problems can occur when the two companion servers cannot figure out how to talk to each other. In my general experience, these problems tend to fall into the broad general areas of connectivity issues, Windows server issues, password and user profile issues, and application authority issues. The interesting thing about this list is that when two servers have trouble talking, you can generally start at the bottom of the list and quickly work your way up to the level where the problem actually exists. Let’s look at each issue in turn and see how you can solve your cross-server issues.

Connectivity Issues

Connectivity issues may be the easiest problem to diagnose. All communication issues come down to the simple question of whether or not your servers can find each other over a TCP/IP connection. When IP communications fails, here are the obvious things to check on your i5 server and on your companion server.

Are the IP interfaces up and transmitting on both machines? You can check this on your i5 box by using the following utilities (you may also need to check these items on your companion servers but for the purposes of this article, we are only looking at the i5 configuration).

The first step is to see if basic communication is open between the two servers. This can be established through the use of a simple PING command. Try pinging your companion server by using both the DNS name of the server and the IP address that the server resides at. Here’s what a Verify TCP/IP Communications command (PING) might look like when run from a 5250 command line.

PING RMTSYS(‘10.10.10.10’)

Note: PING may not be available in earlier versions of the OS/400 operating system.

The biggest difference between running PING on an i5 box and running it on a Windows box is that on the i5, IP addresses must be enclosed within single quotes (‘xxx.xxx.xxx.xxx’) in the command while IP addresses for a non-i5 PING command are not enclosed in quotes. If you are using PING to check transmission using a DNS name, you would code your PING statement this way.

PING RMTSYS(WWW.Internetaddress.Internetextension)

If you can reach the server through its IP address but not through it DNS name, there may be a problem with your i5’s DNS. You can look up the IP addresses of the DNS servers your i5 box is using by running the Change TCP/IP Domain command (CHGTCPDMN) on a 5250 command line. You can find this same information in iSeries Operations Navigator (OpsNav) by right-clicking on the Network –> TCP/IP Configuration node under your i5 partition and selecting Properties from the pop-up menu that appears. Inside the TCP/IP Configuration Properties panel, click on the Host Domain Information tab and the DNS information will be displayed.

If your PINGs are rejected or never reach your companion server, you can also use the Trace TCP/IP Route command (TRACEROUTE), which was added to the operating system with i5/OS V5R1. TRACEROUTE displays the IP addresses of each hop that a packet travels through to reach its final destination. If you see that the i5’s TRACEROUTE path to its companion server is stopped at a particular IP address, then you can investigate that network or Internet component to see why it’s not forwarding your traffic. i5 TRACEROUTE is similar to its non-i5 counterparts with a few notable exceptions, including:

You can write TRACEROUTE output to an i5 data queue, which other i5 and non-i5 programs can read to monitor TCP/IP routing.
You can specify that the trace should either use any active IP interface on your server or that it should trace its way between servers by using one specific IP address interface that resides on your i5 server.

TRACEROUTE’s main benefit lies in determining if and where inter-server communication is being stopped by an intermediary machine, such as a firewall or a router.

If your system is unable to ping its partner or use TRACEROUTE, it may be because you are having a problem with your interfaces or your Ethernet lines. To check the status of your Ethernet IP interfaces, type in the Configure TCP/IP Interfaces command (CFGTCP) from a 5250 session and select option 1, Work with TCP/IP interfaces from the CFGTCP menu. The Work with TCP/IP Interfaces screen shows you all the IP addresses that are currently defined on your system. Press F11=Display interface status and the screen will show you whether the interfaces for each IP address are active or not. If your target interface is inactive, you can try to restart it by a typing a 9=Start in front of the interface entry. If the interface still doesn’t start, check the line description it is attached to by entering a Work with Configuration Status command (WRKCFGSTS) to show all your active line descriptions, like this:

WRKCFGSTS *LIN

If the line is varied off, try varying it back on again. If that doesn’t work, you can start troubleshooting your line description to see if there’s a problem with the physical connection to your system i5, iSeries, or older AS/400.

The OpsNav alternative to using CFGTCP to check your interfaces is to open the Network –> TCP/IP Configuration –> IPv4 or Network –> TCP/IP Configuration –> IPv6 nodes on your partition and view, start, and stop your IP interfaces. To work with your Etherline lines using OpsNav, open the Network –> TCP/IP Configuration –> Lines node for your partition.

If your IP interfaces and communication lines are working correctly, you may want to check to see if there are any exit point programs that may be blocking outside access to your system. Many shops have applied user-written exit point programs to control SQL database access, ODBC traffic, and FTP access. Exit point programs sometimes contain tables that list out valid users who can and cannot access the system. Exit point access may also be controlled by an authorization list. If your companion server cannot retrieve data from your i5 box, it may be that the user ID they are using needs to be added to an exit point access list or table to use the system. In recent columns, I have written about Bryan Dietz’ WRKODBCJOB freeware, which helps you designate which users can and cannot access i5/OS data through ODBC, OLE DB, JDBC, .NET and other common connectivity methods. I use Bryan’s product in my shop and whenever new users need access, I have to remember to add them to the product’s table before they can access the system.

Again, also remember that these are basic connectivity checks that you perform on your i5 system. If the problem cannot be solved on the i5 end, you may also need to perform similar connectivity checks on your companion server.

Windows Server Issues

If your companion server is a Windows machine and you have determined that basic i5-to-Windows IP connectivity isn’t stopping the two machines from communicating, here are a few common Windows-to-i5 problems that might help your troubleshooting.

Is the Windows server running out of disk space? Strange things can happen when Windows disk space fills up and depending on how your application software works, your application may just stop processing when Windows disk drives fill up. So when communication problems occur, be sure to perform a quick check on Windows storage.

Are your application log files getting too big? Be particularly wary of application software that produces log files, such as Lotus Notes, running on a Windows server. When a Notes log file becomes too big after running for weeks on end, it can easily fill up disk drives. Log files can also sometimes grow too large for an application.

Last year, I was working with a Windows workflow product that produced log files detailing every single i5 transaction it created. Even though the product’s log file didn’t fill up the server, the log eventually got too big for the application to handle and it stopped communicating with my i5. The solution was to set up an automatic application that every so often shut down the application software, deleted the log file, and then restarted the application. You may also be able to check log file settings to see if you can limit application log file sizes so that the file never gets too big.

Password and User Profile Issues

Another issue that gets in the way of cross-server communication is passwords. Password disablement can take several forms and here are some of the more common problems I have run into.

The user ID and password the companion server use for i5 communication are embedded on the companion server, and the password changes on the i5 box. Many Windows-to-i5 communications require embedded passwords and the password on the Windows box may be out of sync with the user’s actual i5/OS password.
The password may have expired. When an i5 password expires, the operating system may allow a companion server user to sign on but it may then ask the application to change the password, which could stop the application from functioning.
The user ID and password may be disabled because someone else unsuccessfully tried to log on using that profile and your system is set up for automatic profile expiration after a set number of incorrect sign-on attempts. If you are embedding a user profile and password inside a companion server, make sure the password name is a user ID that is not generally known or easily guessable.

One possible way to get around password issues is to configure your Windows server to use a user profile that doesn’t have a password for i5 access, if your software allows it. That could effectively bypass some of these problems.

Application Authority Issues

Another cross-platform communication issue may occur when a server job actually needs to start a third-party i5 application to transfer data. Some older applications require user authorization in order to start and use their package. If the i5 profile that a companion server uses isn’t authorized to that software, you may not be able to complete your server-to-server transaction. So if all else fails in your quest to find your communication failure, check to see if there are any special third-party authorizations that need to be configured in order to make the connection.

About Our Testing Environment

All configurations described in this article were tested on an i5 box running i5/OS V5R3. The iSeries Navigator configuration (OpsNav) was tested by using the OpsNav version that comes with iSeries Access for Windows V5R3M0. However, you may notice variations in pre-V5R3 copies of these features, as well as on iSeries and AS/400 machines running OS/400 V4R5 and below. These differences may be due to incremental command improvements that have occurred from release to release.