Numerical Methods Question
Posted: Tue Feb 15, 2022 3:28 am
I've got a really odd numerical result that I can't explain. Look at the following code snippet -- you can copy and run and you will get the MesageRequester message to see the results.
f1.d = 325.0
f2.d = 0.125
f3.d = f1 + f2
f4.d = f1 + 1.0 / 8.0
f5.d = -1 * f1 + 1.0 / 8.0
f6.d = -1.0 * f1 + (1.0 / 8.0)
MessageRequester ("Values", "f1 = " + f1 + ~"\n" + "f2 = " + f2+ ~"\n" +"f3 = " + f3 + ~"\n" + "f4 = " + f4 + ~"\n" +
"f5 = " + f5+ ~"\n" + "f6 = " + f6 )
The results are F1 = 325, F2 = .0125, F3 and F4 = 325.125 BUT F5 and F6 = -324.875
The calculation should all be in FLOAT and an error of 0.5 is both huge and unacceptable. As an old FORTRAN coder I understand that precision decreases with magnitude and that you have to be careful to keep fractional calculations in FLOAT or DOUBLE to avoid truncation
to integer (Hence 1.0 / 8.0 rather than 1/8 which could be zero). Any thoughts would be helpful. Any helpful thoughts would be appreciated.
f1.d = 325.0
f2.d = 0.125
f3.d = f1 + f2
f4.d = f1 + 1.0 / 8.0
f5.d = -1 * f1 + 1.0 / 8.0
f6.d = -1.0 * f1 + (1.0 / 8.0)
MessageRequester ("Values", "f1 = " + f1 + ~"\n" + "f2 = " + f2+ ~"\n" +"f3 = " + f3 + ~"\n" + "f4 = " + f4 + ~"\n" +
"f5 = " + f5+ ~"\n" + "f6 = " + f6 )
The results are F1 = 325, F2 = .0125, F3 and F4 = 325.125 BUT F5 and F6 = -324.875
The calculation should all be in FLOAT and an error of 0.5 is both huge and unacceptable. As an old FORTRAN coder I understand that precision decreases with magnitude and that you have to be careful to keep fractional calculations in FLOAT or DOUBLE to avoid truncation
to integer (Hence 1.0 / 8.0 rather than 1/8 which could be zero). Any thoughts would be helpful. Any helpful thoughts would be appreciated.