When a 10-Digit Variable Won't Hold a 10-Digit Number

Hey, Ted:

I've noticed that you and others sometimes use the i and u data types in your RPG IV programs, so I decided to use them, too. Things were going fine until I tried to store a 10-digit stock number in a 10-digit integer field. The program canceled, with message RNQ0103. (The target for a numeric operation is too small to hold the result.) Should I just stick with good old reliable packed decimal?

--Mitch

In this case, yes. You might assume that the maximum value of a 10-digit integer variable would be 9,999,999,999, but that would not be true. The maximum value for a 10-digit integer variable is 2,147,483,647. For a 10-digit unsigned variable, the maximum value is 4,294,967,295. You can find this information in the RPG IV reference. Click here for information about integers and click here to learn about unsigned fields. So a 10-digit integer or unsigned variable isn't going to hold a value like 8,042,213,839.

Your problem relates to an API question that frequently comes up. So let me explain how integer and unsigned variables are stored, then I will address the API issue and hope that those who have had trouble with APIs will better understand why their calls to APIs ended abnormally.

Integer and unsigned variables are stored in the binary (base 2) number system, which has only two digits--0 and 1. Each position in the number represents a power of two. The right-most digit represents two to the zeroth power, or one. The preceding digit represents two to the first power, or two. The digit preceding that one represents two to the second power, or four. And on it goes. Therefore, the binary number 101 means (1 * 4) + (0 * 2) + (2 * 1), or 5.

A 5-digit integer or unsigned variable consists of two bytes of eight bits (binary digits), or 16 bits. The bits represent the following numbers:

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
| | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | + -      1
| | | | | | | | | | | | | | + - -      2
| | | | | | | | | | | | | + - - -      4
| | | | | | | | | | | | + - - - -      8
| | | | | | | | | | | + - - - - -     16
| | | | | | | | | | + - - - - - -     32
| | | | | | | | | + - - - - - - -     64
| | | | | | | | + - - - - - - - -    128
| | | | | | | + - - - - - - - - -    256
| | | | | | + - - - - - - - - - -    512
| | | | | + - - - - - - - - - - -  1,024
| | | | + - - - - - - - - - - - -  2,048
| | | + - - - - - - - - - - - - -  4,096
| | + - - - - - - - - - - - - - -  8,192
| + - - - - - - - - - - - - - - - 16,384
+ - - - - - - - - - - - - - - - - 32,768

If you add all those numbers, you'll see that the sum is 65,535, the largest value a five-digit unsigned variable can hold. Another way to calculate this value is to raise 2 to the 16th power and subtract 1. This is the value of a variable when all bits are set to 1. The value when all digits are zero is zero, of course.

A 10-digit unsigned variable consists of four bytes of eight bits, or 32 bits. Therefore, the maximum value for a 10-digit unsigned variable is 2 to the 32nd power, minus 1, or 4,294,967,295.

Integer variables differ, in that the first digit is reserved for use as a sign bit. If the first bit is zero, the value is positive. If the first digit is 1, the value is negative. This form of storing numbers is known as two's complement notation. The value of a negative number is formed by reversing all the bits of the corresponding positive number and adding 1. Since the first bit of a 10-digit integer variable is reserved for the sign, there are only 31 bits available to store the number. Therefore, the greatest value of a 10-digit integer variable is 2 to the 31st power, minus 1, or 2,147,483,647.

Integer and unsigned data types are relatively new additions to the RPG language. Programming languages designed for mathematical and scientific use, such as FORTRAN and C, have had them for years. But that does not mean RPG had no support for binary data. Even when I was learning to program on a System/3, RPG had a binary data type.

RPG binary variables are like integer variables, in that they are stored in two's complement form. However, variables of RPG's binary data type are handled in the same manner that packed and zoned decimal variables are handled; that is, they can have implied decimals and defined maximum lengths. A five-digit binary variable is stored in four bytes and can have a maximum value of 99999.

This is where many programmers make a mistake. When they read that an API requires a four-byte binary parameter, they define the parameter as four-digit binary, rather than 10-digit integer. I don't have a lot of time to help people debug their programs, so when people ask me if I've had trouble with a certain API, the first thing I do is explain how to define the binary parameters. That often fixes the problem.

In ancient times, the binary data type was sometimes used to conserve disk space. As a rule, there is little or no reason to use the binary data type in RPG.

To get back to your original question, I use integer and unsigned variables for "non-data" purposes, such as loop control variables and counters. I also use them when calling APIs that demand them. Otherwise, I stick with good old reliable packed and zoned decimal.

--Ted

Sponsored By INFINIUM USERNET

Infinium UserNet is the user-run organization for professionals who support, manage and rely on Infinium software applications. Attend the 2003 Infinium UserNet Conference & Expo from June 18-20 in Oak Brook, IL. Technical education, in-depth seminars, user experiences, new product showcase and networking opportunities abound. For the latest information and registration, visit www.infiniumusernet.org