Page 1 of 1
ValF() Issue
Posted: Mon Feb 18, 2008 1:50 pm
by PB
These are bugs, surely?
Code: Select all
n$="1234567890"
Debug ValF(n$) ; Returns 1234567936.0
Code: Select all
n$="1234567890"
n.f=ValF(n$)
Debug n ; Returns 1234567936.0
Posted: Mon Feb 18, 2008 2:00 pm
by Deeem2031
Try doubles if floats are too inaccurate for you. No bug.
Posted: Mon Feb 18, 2008 2:06 pm
by PB
Yes, doubles work, but 1234567890 isn't a large number, so I would've thought
that it'd fit easily into a float? How many digits is a float accurate too, then?
Is it best just to ditch floats altogether and just rely on doubles for everything?
Because that's how it comes across.
Posted: Mon Feb 18, 2008 2:14 pm
by freak
Floats have a precision of about 7 decimal digits, doubles have a precision of 16 digits.
Posted: Mon Feb 18, 2008 8:57 pm
by PB
But there's no decimal points in 1234567890, though. Well, to be technical
it's 1234567890.0 then, but that's only 1 decimal point. What am I not
understanding here?
Posted: Mon Feb 18, 2008 9:32 pm
by rsts
He might mean digits, not points.
cheers
Posted: Mon Feb 18, 2008 11:36 pm
by Demivec
PB wrote:But there's no decimal points in 1234567890, though. Well, to be technical
it's 1234567890.0 then, but that's only 1 decimal point. What am I not
understanding here?
If there is precision to only 7 digits the 123456
7890.0 shows which digit is the last accurate one. So if you convert the string "1234567890" you will get precision in the first 7 digits and the rest are less precise (garbage)
1234567936.0 . The less precise ones are due to limitations converting binary to decimal, i.e. 4 binary digits %0100 are only accurate to 1 decimal digit because they represent values of 0 to 15. So can only represent the first decimal place accurately (0 to 9), the second decimal place can only be represented from 0 to 5 and so it is inaccurate. If you increase the # of bits, and include an exponent you can see why there is a limitation with the floats and doubles.
Posted: Tue Feb 19, 2008 8:31 am
by PB
> He might mean digits, not points
Ah, that's where my confusion comes from. Thanks for clarifying!
Now for all my apps:
Search: .f
Replace: .d
