Johnny Can’t Read and RPG Can’t Do Math
October 25, 2002 Timothy Prickett Morgan
I ran into an anomaly and was wondering if you could shed some light on it.
A subprocedure that I wrote doesn’t return the correct value. The problem is caused by an expression that doesn’t return the value that I think it should.
Why does the expression, which returns the correct value (.3) in the Test4 evaluation, apparently return an incorrect value (.29999) when combined in another expression?
H EXPROPTS(*MAXDIGITS) D X S 5 0 Inz(1) D Y S 5 0 Inz(6) D ABC S 15A Inz('13077.39') D NUM S 15 5 Inz(0000013077.00000) D Digits S 10 Inz('0123456789') D Test1 S 15 5 D Test2 S 15 5 D Test3 S 15 5 D Test4 S 15 5 D Rslt1 S 15 5 D Rslt2 S 15 5 /Free Test1= (.1**X); Test2= (%Scan(%Subst(%Trim(ABC):X+Y:1):Digits)-1); Test3= (Test1)*(Test2); Test4= (.1**X)*(%Scan(%Subst(%Trim(ABC):X+Y:1):Digits)-1); Rslt1= NUM + (Test1)*(Test2); Rslt2= NUM + (.1**X)*(%Scan(%Subst(%Trim(ABC):X+Y:1):Digits)-1); *InLR= *On; /End-Free
Running the program in debug produces the following results:
TEST1 = 0000000000.10000 TEST2 = 0000000003.00000 TEST3 = 0000000000.30000 TEST4 = 0000000000.30000 RSLT1 = 0000013077.30000 RSLT2 = 0000013077.29999
RSLT1 and RSLT2 should yield the same value but don’t.
The problem does not seem to be precision-related, since the incorrect value requires a precision of 5 decimal places to produce and the correct value is only 1 decimal place. I tried both EXPROPTS(*MAXDIGITS) and EXPROPTS(*RESDECPOS) with no effect.
Welcome to the wonderful world of floating point arithmetic, Paul. Your problem reminds me of some of the war stories I used to hear from old timers when I was just getting into this business. Before COBOL was introduced, businesses were using FORTRAN. Rounding errors were an everyday occurrence.
Here’s some information I received from Barbara Morris of the RPG compiler development team at IBM‘s labs in Toronto.
Since 1**X is a floating point expression, the whole expression is done in floating point. When you use ** or any floating point value in an expression, you should consider whether you want the whole expression to be done in floating point. If not, use the %inth or %dech built-in functions to return to decimal arithmetic.
Also, half adjust (eval(h) or %inth or %dech) should always be used with floating point, unless truncation is really wanted. Using eval(h) for the expressions in this case gives .30000 all the time.
RJS SOFTWARE SYSTEMS
Implement Document Imaging on your iSeries-AS/400 in 30 minutes or less
Image Server/400 is a Web browser-based document image management system for the iSeries.
Documents can be quickly scanned and stored in IFS folders, and then located and retrieved for viewing via any Web browser. Integrate with other iSeries-AS/400 applications.
Visit us at COMMON, Booth 418, call us at 888-RJS-SOFT, or download a FREE fully functional demo from our Web site at