Guru: Watch Out For This Pitfall When Working With Integer Columns

May 22, 2023 Gregory Simmons

Remember that awesome jungle game where you had to guide the hero through a series of increasingly hard obstacles to gather treasure? Jumping over snakes, scorpions, and rolling logs, swinging on vines over alligators –your timing had to be just right. Pitfall Harry survived in a world of 255 screens, each of which were 160 x 192 pixels and a dazzling 128 colors! Well today, I want to make you aware of a pitfall which caught me off guard a while back. I hope this article will help you avoid my pitfall.

There I was, writing my code, happy as a lark, when all of a sudden, testing brought a bug to my attention. When I debugged and tracked down the issue, I was astonished to find that my integer field was topping out at 999,999,999 instead of the expected 2,147,483,647. I will demonstrate the issue and how to correct it. Just as the beloved Atari game Pitfall from my youth, this trap has three alligators or gotchas to be aware of and avoid.

So, let’s create a quick table to play with.  This is a simple table which just contains three columns of the three types of integers

Create or Replace Table My_Int_File For System Name MyIntFile (
My_Small_Int For MySmallInt SmallInt Default 0 Not Null,
My_Int       For MyInt      Int      Default 0 Not Null,
My_Big_Int   For MyBigInt   BigInt   Default 0 Not Null
)
RcdFmt rMyIntFile;

This table above should have three fields, capable of holding data values as such:

Now, let’s write a small RPG program to experiment with these integer columns. For this example, I will just play with the myInt column.

1      **free
2      Dcl-ds dsMyIntFile Extname('MYINTFILE') Qualified End-ds;

3      Dcl-s myInt10 Int(10);

4      dsMyIntFile.myInt = *hival;
5      myInt10 = *hival;

6      *InLr = *On;
7      Return;

Line 1: **Free – We’ll be going 100 percent RPG free for this program.

Line 2: I’ve defined a data structure like the file that we just created.

Line 3: For later comparison, I’ve created a standalone variable of type Int(10).

Line 4: Assign *hival to the field column in my data structure, mySmallInt.

Line 5: Assign *hival to the stand alone variable, myInt10.

Now for the interesting part. When I run this program and stop it in debug on line 6 and add my two fields to the monitors tab in the RDi debugger perspective. This is what I get:

Did any of you cock your head to the side as I did when I first saw this? Why didn’t dsMyIntFile.MyInt yield the same as MyInt10, 2147483647? After some head scratching and research, I discovered that by default, RPG treats integer fields in externally described files and data structures as BINDEC or Binary Decimal fields.

Thankfully, this is an easy fix. There is a control option available to make integers in externally described files and data structures to behave the same as internally defined standalone variables. So, one small alteration to my test program:

1      **free
2      Ctl-Opt ExtBinInt(*Yes);

3      Dcl-ds dsMyIntFile Extname('MYINTFILE') Qualified End-ds;

4      Dcl-s myInt10 Int(10);

5      dsMyIntFile.myInt = *hival;
6      myInt10 = *hival;

7      *InLr = *On;
8      Return;

Line 2: The ExtBinInt(*YES) control option allows your program to make use of the full range of numbers for that type of integer.

Now, when I rerun my program in debug, these are the values I get:

All better! Ever since this discovery, I use this control option in all of my RPG programs to keep me protected from this pitfall.

Until next time, happy coding.

Gregory Simmons is a software engineer with PC Richard & Son. He started on the IBM i platform in 1994, graduated with a degree in Computer Information Systems in 1997 and has been working on the OS/400 and IBM i platform ever since. He has been a registered instructor with the IBM Academic Initiative since 2007, and holds a COMMON Application Developer certification. When he’s not trying to figure out how to speed up legacy programs, he enjoys speaking at COMMON and other technical conferences, running, backpacking, SCUBA diving, hunting, and fishing.