Johnny Can’t Read and RPG Can’t Do Math

October 25, 2002 Timothy Prickett Morgan

Hey, Ted:

I ran into an anomaly and was wondering if you could shed some light on it.

A subprocedure that I wrote doesn’t return the correct value. The problem is caused by an expression that doesn’t return the value that I think it should.

Why does the expression, which returns the correct value (.3) in the Test4 evaluation, apparently return an incorrect value (.29999) when combined in another expression?

H EXPROPTS(*MAXDIGITS)
D X               S              5  0 Inz(1)
D Y               S              5  0 Inz(6)
D  ABC            S             15A   Inz('13077.39')
D  NUM            S             15  5 Inz(0000013077.00000)
D  Digits         S             10    Inz('0123456789')
D   Test1         S             15  5
D   Test2         S             15  5
D   Test3         S             15  5
D   Test4         S             15  5
D   Rslt1         S             15  5
D   Rslt2         S             15  5

 /Free
      Test1= (.1**X);
      Test2= (%Scan(%Subst(%Trim(ABC):X+Y:1):Digits)-1);
      Test3= (Test1)*(Test2);
      Test4= (.1**X)*(%Scan(%Subst(%Trim(ABC):X+Y:1):Digits)-1);
      Rslt1= NUM + (Test1)*(Test2);
      Rslt2= NUM + (.1**X)*(%Scan(%Subst(%Trim(ABC):X+Y:1):Digits)-1);
      *InLR= *On;
 /End-Free

Running the program in debug produces the following results:

TEST1 = 0000000000.10000
TEST2 = 0000000003.00000
TEST3 = 0000000000.30000
TEST4 = 0000000000.30000
RSLT1 = 0000013077.30000
RSLT2 = 0000013077.29999

RSLT1 and RSLT2 should yield the same value but don’t.

The problem does not seem to be precision-related, since the incorrect value requires a precision of 5 decimal places to produce and the correct value is only 1 decimal place. I tried both EXPROPTS(*MAXDIGITS) and EXPROPTS(*RESDECPOS) with no effect.

Any ideas?

— Paul

Welcome to the wonderful world of floating point arithmetic, Paul. Your problem reminds me of some of the war stories I used to hear from old timers when I was just getting into this business. Before COBOL was introduced, businesses were using FORTRAN. Rounding errors were an everyday occurrence.

Here’s some information I received from Barbara Morris of the RPG compiler development team at IBM‘s labs in Toronto.

Since 1**X is a floating point expression, the whole expression is done in floating point. When you use ** or any floating point value in an expression, you should consider whether you want the whole expression to be done in floating point. If not, use the %inth or %dech built-in functions to return to decimal arithmetic.

Also, half adjust (eval(h) or %inth or %dech) should always be used with floating point, unless truncation is really wanted. Using eval(h) for the expressions in this case gives .30000 all the time.

— Ted

This Issue Sponsored By

Table of Contents

Content archive

Recent Posts

Subscribe

Pages

Search