Date:         Fri, 3 Jul 1998 02:51:30 GMT
Reply-To:     Melvin Klassen <Klassen@UVIC.CA>
Sender:       "SAS(r) Discussion" <SAS-L@UGA.CC.UGA.EDU>
From:         Melvin Klassen <Klassen@UVIC.CA>
Organization: University of Victoria
Subject:      Re: Precision
Content-Type: text/plain; charset="us-ascii"

On Wed, 1 Jul 1998 20:06:04, Myra.Oltsik@responseinsurance.com wrote:
 
> I got a $0.01 difference in a value calculated in MVS
> and a value calculated on the PC (NT). The calculation is:
>
>            ($269.90 / 124) * 31
>
> The PC gives the results as $67.475, rounding to $67.48.
> With MVS, it rounds to $67.47. I want to document the reason for the
difference
> as this data is used for Accounting purposes. The 1 penny difference isn't a
big deal,
> but if an auditor finds it in the future, I want the explanation ready.
 
Using REXX and OS/2 on a desktop system with an Intel Pentium
processor,
I can get many "answers":
 
say (269.90/124)*31
67.4749999
  ................................................ REXXTRY.CMD on OS/2
say 269.90*(31/124)
67.4750
  ................................................ REXXTRY.CMD on OS/2
say (269.90*31)/124
67.475
 
The first result rounds "down" to 67.470000, but the others do not.
 
The "real" problem is that '269.90' cannot be stored without
"round-off" error,
and multiplying by 1/4 (suprisingly!) affects the answer, on some
platforms.
 
Why?  The IBM System/390 stores the number as "some fraction
multiplied
by some power of 16", while other platforms store the number as "some
fraction
multiplied by some power of 2".  On the System/390, dividing by 4
causes
both the "fraction" and the "exponent" to be altered, and one or two
bits of the
fraction are being truncated.  On the power-of-two platform, only the
exponent
needs to be altered, while the fractional-part remains unchanged.