Guru: IBM i Experience Sharing, Case 1 – Object Authority Check And Batch Job Performance
March 7, 2022 Satid Singkorapoom
Batch processes are perennial in virtually all kinds of business processing. From time to time, customers have to deal with batch runs that take too long, and many factors influence run time. One such factor in IBM i is how you assign object authority access rights for user profiles to the libraries and objects that are accessed by batch jobs. The importance of this factor can be found in the following case study.
Many years ago, a customer asked me to determine why batch run time took too long. The customer ran 25 concurrent batch jobs in the batch process every night on a Power7 server with 7 CPU cores, and the entire process took about 8 hours to finish. The customer wanted run time to be substantially shorter, and contacted my local IBM software services team for help.
My services team colleague collected all required performance data and sent it to our world-wide IBM i support team. The support team told us that all performance factors in Power7 server hardware and the operating system looked good (meaning not being utilized beyond their capacity) except for a quite high “object authority check” overhead. (This performance data is in IBM i performance database table QAPMSYSTEM. See https://www.ibm.com/docs/en/i/7.1?topic=data-collection-services-files-qapmsystem for more information). At the time, no one knew what to do to reduce this overhead, and I was asked to step in to solve the problem.
I produced a performance report and also saw that the server hardware was quite powerful and generally underutilized during batch run period, as originally observed by our world-wide support team. But this was the first time I had ever encountered this object authority check overhead factor and its effect on batch run time.
Fortunately in the early days of my IBM career, I had delivered IBM i customer training through a standard class named AS/400 Security Concept and Planning multiple times, and it took me only a short while to recall the following diagram explaining IBM i search order for resolution of object access rights of a job’s user profile to an object. The search stops as soon as it can determine a clear access right that allows or rejects the manner of access (read, update, add, delete, run, etc.).
The diagram shows that IBM i determines whether or not an object access rights assignment can be found in the user profile that runs the job first, starting at whether there is *ALLOBJ special authority in that profile or not. If not, it proceeds to the next step. If no concrete access right assignment is found (allowed or rejected) in the user profile (by private authority or next by authorization list), IBM i proceeds to check the group profile(s) involved (which is actually another user profile object) and then public authority of the object being accessed, and lastly in the adopted user profile, but only if “adopted authority” is used. The deeper IBM i has to traverse down the path in the diagram, the more the processing overhead.
If a job accesses an object only once, traversing the entire path to resolve to a concrete access right does not waste a lot of CPU cycles. For example, if I run a program that adopts authority of another user profile just to retrieve a record from a physical file and no more, then IBM i traverses almost to the bottom of the search path above but this happens only once and there is no worry about the overhead. But in a typical business database processing environment, it is normal for many jobs to run concurrently, each job accessing data records from many physical files, repeatedly opening a file, reading or writing, and closing the file. This process invokes an authority check traversal many times, accumulating authority check overhead. This was the case for my customer of this article as well.
I asked the customer how the user profile that ran the 25 batch jobs at issue was assigned object access authority to the libraries and objects of the core application. I was informed that all 25 jobs ran under one user profile, and access authority was granted by way of a group profile (to which the user profile belonged) and which was specified in an authorization list with proper access rights assigned. Consulting the search path diagram above, you can see that it takes at least seven steps down the entries in the top two rectangles (first for user profile and the next for group profile). I say at least seven steps because if the check in the first group profile of the user (a user profile can be assigned to up to 16 group profiles) does not yield the access right determination, the next group profile is checked (and thus the “loop” indication for the second rectangle in the diagram) and more overhead is incurred in the object authority check process.
When I asked the customer, they told me that they used a group profile in an authorization list to emulate what was implemented in the core application for interactive user profiles. They were not aware of this object authority check overhead.
At this point, I hope we can now see that to reduce this overhead, the authority search path diagram tells us that we should either assign *ALLOBJ special authority for the user profile that runs the batch jobs (which incurs the least amount of the overhead) or assign the user profile with proper access rights directly to all application libraries and objects (“Private authority” entry in the diagram). Using *ALLOBJ is normally not allowed in many companies, as it violates prevailing IT security policy, but I will discuss later about how you may use this method without such a violation.
I told the customer that they could determine the best case scenario of the run time improvement by first trying *ALLOBJ and then private authority. They could compare the results of the two methods and decide which best fit their case. The customer later informed me that the trial run of *ALLOBJ yielded a run time of about 4.5 hours (down from 8 hours), which was quite a dramatic revelation for all involved, including myself. We all recognized the importance of this factor right away, and I have never forgotten it.
After this case, I was involved in similar problem resolution support for many more customers.
Having learned about this case study, you may wonder why we need to use group profiles and authorization lists? Let’s see why.
When business applications are not of the GUI-based, client-server type, all application users run their own non-batch jobs in their own telnet sessions, and there can be hundreds, if not thousands, of such users active on an IBM i server every day. Assigning *ALLOBJ or private authority is impractical, even impossible, to implement. Therefore, group profiles and authorization lists help system/applications administrators deploy less-complicated, and therefore more efficient, object access rights control to all “interactive” users on the server.
Using a group profile lets you give the same data access rights to a number of user profiles. Authorization lists allow you to treat a number of objects under the same set of data access rights assignments, as demonstrated below.
As an example of the benefit you can get, consider a server with a hundred interactive application user profiles. It is possible to deploy, say, 30 group profiles and 15 authorization lists to assign access rights for all these users to all application objects, and there are fewer security-related entities to manage. This concept exists in virtually all major server operating systems because it helps simplify system administration of these users of non-batch jobs.
But you may ask, what about the object authority check overhead with group profiles and authorization lists for these interactive users? Unlike batch jobs, interactive jobs tend to be a series of relatively short run-time sessions, running programs that access smaller amounts of data from fewer application objects than batch jobs, which means object authority check overhead can therefore be relatively much lower.
On the contrary, a typical customer would need only a few user profiles to run their batch jobs, which tend to access much more data from many more application objects. So, there is less need to use group profiles and authorization lists in this case.
I hope you now appreciate that using a different object authority assignment approach for interactive job users than for batch job users would help you ensure minimum run time overhead for the latter.
One last note on *ALLOBJ. I mentioned earlier that it typically violates IT security policy to have more than one or two user profiles in your system with this special authority. But if you are in dire need of *ALLOBJ because the best run time of your batch process is of utmost importance to your business operation, what could you do that will not violate the security policy? I have an idea to share for your consideration.
Basically, you should create a user profile with *ALLOBJ authority. Ensure this profile cannot be used to log on to your system (a disabled profile cannot run interactive jobs but it can run batch jobs), and that no one can use this profile from the Submit Job (SBMJOB) command or by a profile swap API. This profile is used exclusively to run the batch jobs you need and no others. Specify this profile to run all your important batch jobs from a set of “launching” programs that use adopted authority.
In addition to special authority of *ALLOBJ, specify the following profile parameters:
STATUS = *DISABLED
PASSWORD = *NONE
INLPGM = *NONE
INLMNU = *SIGNOFF
No one will be able to sign onto your system with this profile. Let’s call this profile SUPERMAN.
Use the Edit Object Authority (EDTOBJAUT) command to assign “exclusionary” access right to this SUPERMAN user profile object so that its PUBLIC authority is set to *EXCLUDE, and remove all its private authority entries. Change the owner of SUPERMAN to QSECOFR. (There’s no need for this if you use QSECOFR to create SUPERMAN.) This prevents anyone (except for users with *ALLOBJ) from using the SBMJOB command or the profile swap API to use SUPERMAN to run any other jobs.
Identify all the program objects that launch all batch jobs (or compile new programs) and change the owner of these programs to SUPERMAN. Also change the program object’s USRPRF parameter to *OWNER (the default value is *USER). If these programs use the SBMJOB command to launch batch jobs, change the USER parameter of SBMJOB to SUPERMAN. If the programs use the profile swap API, specify SUPERMAN as the user name to swap to.
Lastly, when the batch jobs run, use the Work with Active Jobs (WRKACTJOB) command to ensure that they are running under SUPERMAN. If not, you may need to check the job log to find out what went wrong.
This is just one possible way, and I know that some customers cannot adopt this method for practical reasons, such as when they do not have source code and therefore cannot modify the programs. But I trust you get the idea and can find a proper approach for your case.
User profiles of interactive and batch jobs need to be deployed differently in order to optimize the performance of the latter. I’m confident that understanding this difference will pay handsome dividends for those who value the performance of their batch processes.
Satid Singkorapoom has 31 years of experience as an IBM i technical specialist, dating back to the days of the AS/400 and OS/400. His areas of expertise are general IBM i system administration and performance of server hardware, the operating system, and database. He also has an acquired taste for troubleshooting problems for his IBM i customers in the ASEAN geography.