Admin Alert: Preventing Multiple IPs from Stopping Internet Traffic

Published: October 15, 2008

by Joe Hertvik

We recently added a second IP interface to one of our i5/OS partitions. After an IPL, the partition stopped talking to Internet-based trading partners. It stopped trading FTP files with outside vendors, it wouldn't relay email, and it generally stopped talking to the outside world. This week, I'll look at the common problem that took down our Internet communications and how you can stop it from happening in your shop.

A Case Study in Configuring for Multiple IPs

After the configuration change, we had two IP interfaces on our partition: one for all our normal IP traffic (10.3.1.55); and a second interface that we used for communicating with our Capacity BackUp system (10.3.1.60). This configuration worked fine for awhile but after IPLing the system one day, our System i model 550 suddenly stopped talking to business partner machines outside the local area network.

Here's what happened and how we fixed it through TCP/IP routing entries. If you have multiple interfaces on your partitions, it's worth looking at this case study, as the problem is easy to develop. Fortunately, the fix is universal for most iSeries, System i, and Power i shops.

Each of our interfaces was configured on a Class B subnet with an identifier of 10.3, meaning that its subnet mask was 255.255.0.0. Any traffic configured for a 10.3.x.x address was considered to be on the local network and did not need to be routed through our gateway to the outside world. We verified this by viewing our IP interfaces through option 1 (Work with TCP/IP Interfaces) from the Configure TCP/IP (CFGTCP) command. Our TCP/IP interface descriptions looked like this:

                         Work with TCP/IP Interfaces
                                                            System:
Type options, press Enter.
  1=Add   2=Change   4=Remove   5=Display   9=Start   10=End

     Internet         Subnet              Line      Line
Opt  Address          Mask             Description  Type

     10.3.1.55       255.255.0.0       ETHERLINE    *ELAN
     10.3.1.60       255.255.0.0       ADMINLINE    *ELAN
     127.0.0.1        255.0.0.0        *LOOPBACK    *NONE

For Internet traffic outside our network, we had configured a default TCP/IP route that looked like the following. We viewed the default route setup under option 2 (Work with TCP/IP routes) off the CFGTCP menu.

                            Work with TCP/IP Routes
                                                             System:
 Type options, press Enter.
   1=Add   2=Change   4=Remove   5=Display

      Route            Subnet           Next             Preferred
 Opt  Destination      Mask             Hop              Interface

      *DFTROUTE        *NONE            10.3.1.1        *NONE 

Using this default route entry (*DFTROUTE), i5/OS and OS/400 use the following steps to send traffic outside the local area network to a foreign IP address:

• All traffic destined to go outside the network is sent to its next Internet routing server (the Next Hop value, 10.3.1.1 in this case) through the preferred IP interface listed under the default route (*DFTROUTE).
• When you use *NONE as *DFTROUTE's preferred interface, the traffic will automatically be routed to its next hop through the first active TCP/IP interface on our network.

There was no problem with routing traffic to our outside business partners when we only had one TCP/IP interface (10.3.1.55) because the Internet traffic was always routed using the 10.3.1.55 interface. For security reasons, our business partners and network equipment were configured to only accept and route traffic from 10.3.1.55.

Since small changes can introduce chaos, we created an Internet routing problem when we added a second interface to the partition (10.3.1.60). Since our *DFTROUTE entry says that Internet traffic will always be routed through the first active interface in the network, we could no longer guarantee that traffic would always be routed through the 10.3.1.55 interface. If for any reason the 10.3.1.60 interface became the first active interface, outside traffic to our business partners would be routed through 10.3.1.60 (because the preferred interface is *NONE) and the outside business partners will start rejecting our traffic. In order to be accepted, all Internet traffic for the business partners always had to be routed through 10.3.1.55.

We had two choices for how to make sure traffic was always routed through the correct interface:

1. We could set up our system so that 10.3.1.55 was always the first active TCP/IP interface that came up when the partition started. This would be done through the system startup program where we would activate all TCP/IP interfaces manually as the system IPLs. This works in theory, but there's always the danger of something going wrong if for any number of reasons, the 10.3.1.60 interface comes up before 10.3.1.55 and the .60 IP becomes the first active interface. While starting TCP/IP interfaces in series is a valid idea, it will probably lead to a problem every once in a while when the 10.3.1.60 interface comes up before 10.3.1.55.

Or . . .

2. We could change the routing table so that the *DFTROUTE entry would always use the 10.3.1.55 IP address when routing traffic outside the local area network. By doing this, vendor traffic will always come from the proper interface and there wouldn't be any issues with vendors refusing traffic from 10.3.1.60.

We decided to change the *DFTROUTE entry. We made the change by performing the following steps:

• Recording the current *DFTROUTE routing entry values, in case we had to restore that entry later on.
• Deleting the *DFTROUTE entry on the Work with TCP/IP Routes screen by placing a '4'=Remove in front of the entry and pressing ENTER.
• Adding a new *DFTROUTE entry by running the following Add TCP/IP Route command (ADDTCPRTE).

                         Add TCP/IP Route (ADDTCPRTE)

Type choices, press Enter.

Route destination  . . . . . . . > *DFTROUTE
Subnet mask  . . . . . . . . . . > *NONE
Type of service  . . . . . . . .   *NORMAL       *MINDELAY,*MAXTHRPUT
Next hop . . . . . . . . . . . . > '10.3.1.1'
Preferred binding interface  . .   '10.3.1.55'
Maximum transmission unit  . . .   *IFC          576-16388, *IFC
Route metric . . . . . . . . . .   1             1-16
Route redistribution . . . . . .   *NO           *NO, *YES
Duplicate route priority . . . .   5             1-10

This command can also be reached by using option '1'=Add off the Work with TCP/IP Routes screen.

Once I replaced the *DFTROUTE entry, my TCP/IP routes now looked like this:

                            Work with TCP/IP Routes
                                                             System:
 Type options, press Enter.
   1=Add   2=Change   4=Remove   5=Display

      Route            Subnet           Next             Preferred
 Opt  Destination      Mask             Hop              Interface
      *DFTROUTE        *NONE            10.3.1.1        10.3.1.55

By making this change, all outside Internet traffic would always be routed through the 10.3.1.55 interface. Since my network and the partners were set up to only accept traffic from 10.3.1.55, this solved all our problems with rejected traffic. And because this configuration is dependent on the TCP/IP Route entries rather than the order in which our interfaces come up, we can count on it working correctly every time that we IPL our system or start and stop the interfaces.

Finally, you should note that you don't have to exclusively use a 5250 green-screen session to make these changes. You can also view your IP interfaces and change your TCP/IP routes by using iSeries Navigator (OpsNav). To view and work with your IP interfaces in OpsNav, open the Network→TCP/IP Configuration→IPV4→Interfaces node under your i5/OS partition. To view and work with your TCP/IP routes, open the Network→TCP/IP Configuration→IPV4→Routes node. In some ways, OpsNav is easier to understand than using Option 2 under Work with TCP/IP Routes. This is because not only does it show your default route and any other TCP/IP routes that you have configured on your system, it also shows all the other implicit direct routes that i5/OS will use to route traffic directly to locations inside your local area network.

About Our Testing Environment

Configurations described in this article were tested on an i5 550 box running i5/OS V5R4. Many of the commands are also available in earlier versions of the operating system running on iSeries or AS/400 machines. If a command is present in earlier versions of the i5/OS or OS/400 operating systems, you may notice some variations in the pre-V5R4 copies of these commands. These differences may be due to command improvements that have occurred from release to release.

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot