IBM Patches Security Flaw in OS/400 V5R3

Corrected: January 16, 2007

by Alex Woodie

IBM issued several integrity PTFs last September to fix a security vulnerability in OS/400 and i5/OS V5R3 and V5R3M5. The problem, called the OS/400 Connection Reset Denial of Service Vulnerability, can be exploited by hackers to reset established TCP connections on iSeries and System i servers, according to security firm Secunia, which gave the vulnerability a "less critical" rating.

IBM issued four Program Temporary Fixes (PTFs) on September 26 to fix the problem, in which an established TCP connection can be reset by sending a specially crafted TCP packet. It appears that a hacker could potentially use this technique to launch a denial of service (DoS) attack by repeatedly resetting the connection, thereby forcing a user to disconnect the server's network connection before the DoS attack causes the server to overload and crash.

IBM first included the fix in OS/400 V5R4, Jim Herring, director of System i product management and business operations, said today. "Our guys said it would take an awful lot of work to be able to exploit this exposure, so we decided to fix it first in the V5R4 base code, which was in development at the time, because it would get the highest amount of testing," he said. IBM then applied the fix to V5R3 and V5R3M5 and released the integrity PTFs.

IBM released two Authorized Program Analysis Reports (APARs) including MA33860 and MA33861, which referenced four patches: R530 MF39879 7016 and R530 MF39880 7016 for OS/400 (i5/OS) V5R3, and R535 MF39909 7016 and 535 MF39910 7016 for V5R3M5. MF39879 has since been superceded by MF40178, and MF39909 has been superceded by MF40861.

According to the Secunia advisory posted Monday, the OS/400 security vulnerability is related to the TCP Reset Vulnerability that was first reported by security researcher Paul Watson in April 2004. At the time, there was great concern that the vulnerability could be exploited to launch a massive attack that would cripple the Internet. As it turns out, those fears were largely unfounded. Network equipment vendors, led by Cisco Systems, updated their wares to fix the problem.

Apparently, the problem went unpatched in OS/400 and the new i5/OS operating system for more than two and a half years. Herring said IBM was notified that OS/400's TCP/IP stack was at risk to the exposure, but it's unclear if any iSeries or System i users were hit by DoS attacks. In any event, iSeries and System i users should take the problem seriously and apply the integrity PTFs as soon as possible, if they haven't already done so.

Herring said there are no plans to issue PTFs to fix the problem in previous releases of OS/400.

Security vulnerabilities like this are a rare occurrence for OS/400, which is widely regarded to be one of the most--if not the most--secure operating systems in use. While it's not in any danger of becoming like every hackers' favorite target, Microsoft Windows, anytime soon, IBM OS/400 does occasionally make news with a vulnerability.

Also in November, Secunia reports IBM issued MF33249 to fix the "osp-cert Fix ASN.1" vulnerabilities in its ASN.1 parser for OS/400 V5R3. Secunia gave the vulnerabilities a "moderately critical" rating, one step above the rating it gave the Connection Reset DoS vulnerability.

OS/400 is not without its weaknesses--especially when it comes to implementing standards-based protocols that turn out to have security holes. But when properly configured, OS/400 is practically hacker proof. Its highly regimented access controls make it very difficult for a hacker who's unfamiliar with the system to break it, and its object oriented design make it highly resistant to conventional viruses. In fact, there has never been a documented virus afflicting OS/400 (although security researchers say it's not impossible to create one).

Unfortunately, while security is one of OS/400's strengths, many companies don't take the time to properly configure their server's security settings--either from lack of time and knowledge or a mistaken reliance on the box's security capabilities--leaving them open to problems down the road. For a sobering look at the slipshod approach to security at many OS/400 shops, check out our story on security software developer PowerTech's most recent state of OS/400 security report.

This story has been corrected. IBM issued the integrity PTF in September 2006, not on January 13, 2007, as the story first stated. On January 13, IBM updated the advisory concerning the PTF and the vulnerability it fixed. IT Jungle regrets the error.

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot