Admin Alert: And /QOpenSys and /QOpenSys and /QOpenSys and. . .
Published: April 15, 2009
by Joe Hertvik
Last month, I wrote about an i5/OS issue involving the AS/400 Integrated File System. When opening the /QOpenSys folder inside the AS/400 IFS, I found a second /QOpenSys entry. Opening that entry revealed another /QOpenSys folder, and then another, and another, until the trail ended 15 /QOpenSys entries later. I passed the problem on to the Admin Alert readership for research. This week I explain what they found.
Reopening the Matryoshka Doll
The issue, where one /QOpenSys entry is nested inside another like a Power i Matryoshka doll, is simple to explain. When opening the /QOpenSys file system on a partition, I was surprised to find and open another /QOpenSys entry, making my current directory /QOpenSys/QOpenSys. That folder also contained a /QOpenSys entry. After opening the second entry, my new IFS location became /QOpenSys/QOpenSys/QOpenSys. Fascinated, I continued following the recursive /QOpenSys entries down the system until I opened the last /QOpenSys entry on my system, which was located 15 layers deep from the root directory (/) of the AS/400 IFS.
According to the Work with Object Links command (WRKLNK) I used to explore these entries, my final iterative /QOpenSys entry looked like this:
Work with Object Links
Directory . . . . :
Type options, press Enter.
2=Edit 3=Copy 4=Remove 5=Display 7=Rename
8=Display attributes 11=Change current directory ...
Opt Object link Type Attribute Text
I also discovered that this same recursive situation existed on four of the other five i5/OS partitions in my shop, so it wasn't something I could pinpoint to a single partition. It was beginning to look like something that was endemic to the i5/OS operating system, itself.
As strange as it sounds, there is also a secondary issue with the Matryoshka problem, in that there's a limit on the number of nested /QOpenSys entries that can be opened inside an AS/400 IFS. Each time I followed the links down to the very last /QOpenSys entry on my system and tried opening that entry, I would receive an error saying "Path name resolution causes looping."
Confused, I threw this situation out to our loyal Admin Alert reader base. Here's what they threw back.
What the Readers Know
Thirteen readers contacted me to let me know that yes, indeed, they saw the same issue on their systems. They had limited recursive /QOpenSys entries on i5/OS versions V5R3, V5R4, V5R4M5, and V6R1 systems, and opening the last entry also produces a "Path name resolution causes looping" error. While nobody (including myself) had found any explanation on why this was so common, reader Jeff Green pointed out that IBM states that the /QOpenSys file system is compatible with UNIX-based operating system standards such as POSIX and XPG, and that it supports multiple hard links and symbolic links.
At this point, the reader clues indicated that the recursive /QOpenSys entries could be symbolic links that point to the original /QOpenSys directory. That is, no matter how many /QOpenSys/QOpenSys/QOpenSys/, etc., entries I open, each successive entry may be pointing me back up to the original /QOpenSys directory off the root (/) of the IFS.
With this feedback, I now had a working theory. I only had to figure out how to confirm these suspicions. That solution showed up in my mailbox courtesy of reader Robert Clay. Robert showed me an easy green screen way to determine what type of IFS object link I was looking at with these recursive entries. Here's the drill for determining whether an AS/400 IFS object is a symbolic link and what object the link is pointing to.
1. Run the Work with Object Links (WRKLNK) command for one of your recursive /QOpenSys entries, and set the Detail (DETAIL) parameter to *EXTENDED, so that the command looks something like this:
WRKLNK OBJ('/QOpenSys/QOpenSys') DETAIL(*EXTENDED)
*EXTENDED is a special option that changes the Type field on the WRKLNK display to show additional information about any symbolic links on the display. When I ran the command for the first recursive /QOpenSys entry, it showed me this information:
Work with Object Links
Directory . . . . : /QOpenSys
Type options, press Enter.
2=Edit 3=Copy 4=Remove 5=Display 7=Rename
11=Change current directory ...
Opt Object link Type Attribute Text
The SYMLNK->DIR variable under the Type field confirms that the /QOpenSys/QOpenSys entry is a symbolic link that points to a directory. I tried it on some of the other recursive /QOpenSys entries and they all said the same thing. Each nested /QOpenSys entry was a symbolic link to another directory.
2. Now that I know that all of these entries are symbolic links, my suspicion was that they pointed to the original /QOpenSys directory from the root of the AS/400 IFS. To confirm that suspicion, I once again ran the WRKLNK command shown in the prior step and retrieved the Work with Object Links screen that tells me the recursive entry is a symbolic link to a directory. When run in *EXTENDED mode, WRKLNK also provides a new option, 12=Work with links, that displays information about a symbolic link. When I entered a 12 in front of one of my recursive /QOpenSys entries, I got the following Display Symbolic Link screen.
Display Symbolic Link
Object link . . . . . : /QOpenSys/QOpenSys
Content of Link . . . : .
To understand where the symbolic directory link is pointing to, you have to understand the concept of the single '.' (dot) in the Content of Link field. This dot is referred to as a dot link. When a dot link is present in a symbolic link, it refers to the last child directory in the Object Link field on this screen. So when you open one of the /QOpenSys/QOpenSys/. . . directories and the Content of Link field has a dot link, the AS/400 IFS will resolve its location back to the /QOpenSys directory under the Root (/) of the AS/400 IFS (it does this by attaching the '/' character to the last child directory name, i.e., '/' + 'QOpenSys' = '/QOpenSys'). In other words, all of my nested /QOpenSys symbolic links will actually resolve back to the original /QOpenSys folder off the root of the AS/400 IFS, when I open the links. For the /QOpenSys/QOpenSys/. . . entries we've been discussing, all roads do lead back to the same place: the original /QOpenSys folder.
With this procedure, I confirmed our suspicions that the IFS' recursive /QOpenSys entries all point back to the original /QOpenSys directory that exists off the root of the AS/400 IFS. What's more, it looks like there is only one symbolic link off the original /QOpenSys directory and the multiple /QOpenSys/QOpenSys entries are merely recursive entries caused by opening the same symbolic link over and over again. The key is that no matter how many times I open the /QOpenSys symbolic link in the /QOpenSys directory, I now know that I will always be accessing the same objects in the /QOpenSys folder. There are no duplicate objects in nested folders.
Curing the Problem. . . If You Wish
While no one can explain how these recursive entries came into being, reader Luc Pittoors emailed me about how he solved this issue and deleted the symbolic /QOpenSys entry in his AS/400 IFS. If you decide that you want to remove this entry, here's Luc's drill and my comments.
- First, take a backup of your AS/400 IFS and make sure that you've backed up the /QOpenSys directory. This will ensure that you can recover your original /QOpenSys structure if you make a mistake.
- Luc said that he then deleted the /QOpenSys/QOpenSys/QOpenSys symbolic link in his AS/400 IFS, the third level down from the Root (/) of the AS/400 IFS. At that point, he said that all his other /QOpenSys symbolic links disappeared, including the link at level 2. This makes sense, considering that there is only /QOpenSys symbolic link and if you delete the symbolic link anywhere in the chain, you will still be deleting the one and only link.
Luc also reported that he performed this procedure on 10 out of 16 iSeries machines that had the same problem, and it worked on each partition. So I have one report that deleting the symbolic links works in clearing up the issue.
If you're contemplating performing the same technique on your machines, there are three steps I recommend you take. First, decide if you really need to delete your /QOpenSys symbolic link. If it is not causing any harm, you may just want to leave it on your system rather than risking accidentally disrupting the system during the deletion process. Second, if you decide to delete the link, take a good AS/400 IFS backup as I recommended. Finally, perform the deletion process first on a test partition, if you can, and let it run for a while (a number of days) before attempting the deletion on your production partitions. It's better to see what the long-term effects of the deletion are before you risk doing it on your production partitions, and the /QOpenSys/QOpenSys link could conceivably be used by an applications program.
About the Limited Redundant /QOpenSys Folders
The final question was why the symbolic link recursion ends at 15 folders with the "Path name resolution causes looping" error. I also had reports that 15 folders wasn't the same number that some of my readers were getting. One reader told me that he got the error after opening the link 12 times. So something else was at work here.
Once again, Robert Clay provided a possible answer:
Apparently, there is a limit to the number of times you can do this [link back to another directory]. This information is located in QSYSINC/SYS.LIMITS. Pull it up in PDM and use FIND to browse for POSIX_SYMLOOP. At V5R3 and V 6.1 (the two OS levels that I have available to check), the value is 16. The source shows that this is the invariant value for the "Number of symbolic links that can be traversed in the resolution of a pathname in the absence of a loop."
This may explain why the IFS errored out for me after recursively opening the link 15 times (16 levels deep). I probably violated my limit of 16. If the reader who reported erroring out after 12 recursive links had his POSIX_SYMLOOP value set to 13, this theory also sounds plausible.
QSYSINC/SYS.LIMITS refers to the LIMITS member in the SYS file in the QSYSINC library. You can view this member by using Programming Development Manager (PDM), SQL, or another file access method and looking for the POSIX_SYMLOOP variable. When Robert sent me this tip, he stressed that it was only a guess and it may not necessarily be the correct answer. So even though it jives with the other information I've presented here, please treat this section as a possible answer unless we are able to present further documentation (if ever).
Answers and Questions
So this is what we have from our collective research into this issue.
- It's common to recursively open the symbolic /QOpenSys link in the /QOpenSys directory off the AS/400 IFS. Many other i5/OS shops have run into this situation. It is caused by continually opening the same symbolic link that loops back to the /QOpenSys directory.
- Multiple recursive /QOpenSys opens are not dangerous to your system, since opening any of the links will always direct you back to the real /QOpenSys directory. You can attempt to remove the symbolic link if you want, but it doesn't seem to be a problem to keep it. However, be warned that I have no confirmation if there are any repercussions to removing the link, so perform the removal on a test machine for a reasonable amount of time before attempting it on a production box.
- The number of levels deep that the recursion can go may depend on the POSIX_SYMLOOP value in the LIMITS member of your QSYSINC/SYS file. This is a plausible guess that could be proven wrong with more information, but it jives with our facts.
However, the big question still hasn't been answered. We've been unable to determine what causes the symbolic /QOpenSys link to be created in the first place. If anyone at IBM or elsewhere has this information or would like to expand on the points in this article, please feel free to write in and I'll publish that information in a future article.
Matryoshka Roll Call
Many thanks to the readers who wrote in with information and tips on this issue. I'd like to gratefully acknowledge the following friends of the column who contributed to this article. Thank you very much, and consider yourself no-prized.
Six Ways to Mess Up i5/OS User Profile Security (including the nested /QOpenSys explanation)
Post this story to del.icio.us
Post this story to Digg
Post this story to Slashdot