Johnny Can't Read and RPG Can't Do Math

Hey, Ted:

I ran into an anomaly and was wondering if you could shed some light on it.

A subprocedure that I wrote doesn't return the correct value. The problem is caused by an expression that doesn't return the value that I think it should.

Why does the expression, which returns the correct value (.3) in the Test4 evaluation, apparently return an incorrect value (.29999) when combined in another expression?

H EXPROPTS(*MAXDIGITS)
D X               S              5  0 Inz(1)
D Y               S              5  0 Inz(6)
D  ABC            S             15A   Inz('13077.39')
D  NUM            S             15  5 Inz(0000013077.00000)
D  Digits         S             10    Inz('0123456789')
D   Test1         S             15  5
D   Test2         S             15  5
D   Test3         S             15  5
D   Test4         S             15  5
D   Rslt1         S             15  5
D   Rslt2         S             15  5

 /Free
      Test1= (.1**X);
      Test2= (%Scan(%Subst(%Trim(ABC):X+Y:1):Digits)-1);
      Test3= (Test1)*(Test2);
      Test4= (.1**X)*(%Scan(%Subst(%Trim(ABC):X+Y:1):Digits)-1);
      Rslt1= NUM + (Test1)*(Test2);
      Rslt2= NUM + (.1**X)*(%Scan(%Subst(%Trim(ABC):X+Y:1):Digits)-1);
      *InLR= *On;
 /End-Free

Running the program in debug produces the following results:

TEST1 = 0000000000.10000
TEST2 = 0000000003.00000
TEST3 = 0000000000.30000
TEST4 = 0000000000.30000
RSLT1 = 0000013077.30000
RSLT2 = 0000013077.29999

RSLT1 and RSLT2 should yield the same value but don't.

The problem does not seem to be precision-related, since the incorrect value requires a precision of 5 decimal places to produce and the correct value is only 1 decimal place. I tried both EXPROPTS(*MAXDIGITS) and EXPROPTS(*RESDECPOS) with no effect.

Any ideas?

-- Paul

Welcome to the wonderful world of floating point arithmetic, Paul. Your problem reminds me of some of the war stories I used to hear from old timers when I was just getting into this business. Before COBOL was introduced, businesses were using FORTRAN. Rounding errors were an everyday occurrence.

Here's some information I received from Barbara Morris of the RPG compiler development team at IBM's labs in Toronto.

Since 1**X is a floating point expression, the whole expression is done in floating point. When you use ** or any floating point value in an expression, you should consider whether you want the whole expression to be done in floating point. If not, use the %inth or %dech built-in functions to return to decimal arithmetic.

Also, half adjust (eval(h) or %inth or %dech) should always be used with floating point, unless truncation is really wanted.  Using eval(h) for the expressions in this case gives .30000 all the time.

-- Ted

Sponsored By RJS SOFTWARE SYSTEMS

Implement Document Imaging on your iSeries-AS/400 in 30 minutes or less Image Server/400 is a Web browser-based document image management system for the iSeries. Documents can be quickly scanned and stored in IFS folders, and then located and retrieved for viewing via any Web browser. Integrate with other iSeries-AS/400 applications. Visit us at COMMON, Booth 418, call us at 888-RJS-SOFT, or download a FREE fully functional demo from our Web site at www.rjssoftware.com