Admin Alert: When System i Ethernet Cards Stop Broadcasting

Twice recently my shop ran into an interesting problem with System i Ethernet cards. We were migrating Ethernet line descriptions from one system to another and even though the source system was in restricted mode, the target Ethernet card refused to work correctly until the source system card was unplugged from the network. Here’s what happened and what it teaches us about how System i Ethernet cards work.

Why Migrate Line Descriptions?

Because this situation deals with conflicts caused by migrating configurations, let’s first look at why people migrate Ethernet line descriptions between systems. I can think of three separate situations where an administrator would move or duplicate an Ethernet line description from one system to another.

1. When restoring your operating system to a new machine during a System i upgrade.
2. When performing fail-over processing as you activate a Capacity BackUp (CBU) system to take over processing for a failed System i box or to test your CBU fail over procedure.
3. When restoring a System i configuration on a disaster recovery system.

In all three cases, the end result is that after the migration, you will have two machines containing the exact same Ethernet line descriptions, complete with all the same configuration parameters. Depending on how your lines are configured and what state each machine is in, this situation could prevent the Ethernet cards on both machines from broadcasting correctly in the same network. Here’s what happened in my shop and what could happen the next time you try to perform one of these migrations.

The Deadlock Scenario

I ran into the malfunctioning Ethernet card scenario in two of the situations mentioned above: when we were transferring processing from a production box to a CBU system during a high availability test; and when we were migrating an i5/OS partition from an old i520 box to a new System i 550 machine. In both cases, we were activating an Ethernet line on the new system (the target) based on the same parameters that the line contained on the old system (the source), including the same local adapter address that we had used on the source system. To try to eliminate the possibility of conflicts where the two lines were active at the same time with the same network card parameters, we placed the source System i box in restricted state before we brought up the target system’s Ethernet line.

In both cases, the target system’s Ethernet line froze up like Minnesota in January. We were having trouble connecting to and maintaining a stable Ethernet connection. If we got what looked like a stable connection, it dropped after a few minutes. We puzzled over this problem during the high availability test but it wasn’t until we performed a system migration the next week that we discovered the cause of our Ethernet line drops.

It turned out that the Ethernet card on our source system was still active even though the system was in restricted state. Because of our Ethernet line configuration, the target system Ethernet card was using the same Media Access Control address (MAC address) as the MAC address on the source Ethernet card. Once we unplugged the source system’s Ethernet card, the conflict disappeared because the source system was no longer broadcasting on the network and the target system started behaving perfectly.

What the…?

Now any student of networking knows that two different Ethernet cards aren’t supposed to have the same MAC address. The MAC is a unique identifier that is burnt into every network adapter card ever produced. MACs identify a host system on a network, so that transmitted frames can be delivered to their intended destinations through the Data Link layer (layer 2) of the Open Systems Interconnection Basic Reference Model (OSI Model). On most systems, it isn’t possible to override a NICs MAC address.

The System i isn’t most systems.

On System i boxes (as well as on their predecessor iSeries and AS/400 machines), the MAC can be overridden by specifying a new MAC address in the Ethernet line description’s Local Adapter Address (ADPTADR) parameter. In the Create Line Desc (CRTLINETH) command (Ethernet), ADPTADR is set to *ADPT by default. *ADPT means that when the line is varied on, i5/OS will activate the network card attached to this line with the burnt-in MAC address on the card.

In i5/OS and OS/400 shops, it was (and still is in some cases) common practice to use custom ADPTADR parameters to set up standardized MAC addresses that are referenced by other systems using IBM’s System Network Architecture Distribution Service (SNADS) protocol. SNADS is an operating system application that uses Advanced Program-to-Program Communications (APPC) to talk to and exchange objects with other systems in the SNADS network, and it has been a mainstay of the i5/OS and OS/400 world for decades. As such, even if the system is no longer using SNADS for communications, the custom MAC address may have been configured on the Ethernet line years ago and migrated to the current machine through successive system upgrades.

Given this, it is possible and common to change a System i MAC address and the address will migrate to another System i box when the configuration is restored to a different system. One question remains, however. Why was the Ethernet card with a duplicated MAC address still transmitting when the i5/OS system was in restricted state?

The answer is that the card can remain active even when the system is down. Putting your System i partition in restricted state does not necessarily turn off the card or stop it from transmitting at the OSI layer 2 level. So while the system is restricted, the card can still be broadcasting its last overridden Ethernet address, which can put it in conflict with the same MAC address that was restored and started on the new machine.

Given this situation, you have the following choices to stop the source machine’s card from transmitting the duplicated MAC address while in restricted state. First, you can vary off the Ethernet line on the source machine by using the following Vary Configuration (VRYCFG) command. This will turn off the card on your System i.

VRYCFG CFGOBJ(Ethernet_line_desc) CFGTYPE(*LIN) STATUS(*OFF)

If you need to have the Ethernet cards on your source and target machines active at the same time and both machines use a custom MAC address, you can vary off the line and temporarily or permanently change the ADPTADR address in your source system Ethernet line description. (Note: If you want both systems to transmit at the same time, make sure that you also use a different IP address interface on one of the machines.) Once changed, you can vary the line back on again which will bring up the card with a different custom MAC address that isn’t shared between the two machines.

To change the MAC address, run the following CRTLINETH command on your source machine when the line is varied off.

CHGLINETH LIND(line_description_name) ADPTADR(xxxxxxxxxxxx)

The xxxxxxxxxxxx is a hexadecimal address between ‘020000000000’ and ‘FEFFFFFFFFFF’. The only restriction is that the second digit of the address must contain either a ‘2’, ‘6’, ‘A’, or ‘E’. If you want to make things simpler, you can specify that the network card on the source machine uses the card’s default MAC address (which will not be duplicated on the target line description) by entering the following CHGLINETH command.

CHGLINETH LIND(line_description_name) ADPTADR(*ADPT)

Alternatively, you can also unplug the cable from your network card and that will stop the card from communicating its MAC address to the network. Any of these three techniques will stop your source Ethernet card from interfering with your target machine’s Ethernet transmissions.

Lessons To Be Learned

So putting this all together, we get a slightly different view of how Ethernet cards work on a System i machine and what we can do to avoid network conflicts when transferring configurations from one machine to another. This understanding led me to formulate the following rules of thumb when activating an Ethernet card from a source to a target machine in a migration scenario.

1. Whenever possible, use the burnt-in MAC address of your network card for the Ethernet line description. By doing this, you will ensure that there won’t be any conflicts if and when you need to migrate the source machine Ethernet configuration to a new target box. The MAC address on the target machine will always be separate and unique from the address on the source machine.
2. Understand that a System i Ethernet card can still be transmitting at OSI layer two when its i5/OS partition has been put in restricted mode. As you plan your migration, also make plans to either modify or change any custom MAC addresses on your source machine’s Ethernet line description in order to avoid conflicts with the target Ethernet card. You can do this by using the Create Line Desc (CRTLINETH) command (Ethernet).
3. When all else fails, you can always vary off the Ethernet line or unplug the cable from the Ethernet card on your source System i box. This will ensure that the card won’t interfere with a currently running Ethernet card on the target machine.
4. If you want both systems to be reachable on the network at the same time, make sure that you also provide separate IP address interfaces for each machine, so that you also avoid any IP addressing conflicts in addition to the MAC address conflicts described here.

By understanding how conflict situations occur in a system migration scenario, you can save time and effort when transferring Ethernet line descriptions from one machine to another.

About Our Testing Environment

Configurations described in this article were tested on an i5 550 box running i5/OS V5R3. Most of the commands shown here are also available in earlier and later versions of the operating system running on iSeries or AS/400 machines. If a command or function is present in earlier versions of the i5/OS or OS/400 operating systems, you may notice some variations in the pre-V5R3 copies of these commands. These differences may be due to command improvements that have occurred from release to release.

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot