Just starting out? Need help? Post your questions and find answers here.
gurj
Enthusiast
Posts: 664 Joined: Thu Jan 22, 2009 3:48 am
Location: china
Contact:
Post
by gurj » Wed Mar 22, 2017 1:38 pm
this is bug ?, about g.d of line12
Code: Select all
n=ParseDate("%yyyy.%mm.%dd %hh:%ii:%ss","2016.03.23 17:16:49")
a=ParseDate("%yyyy.%mm.%dd %hh:%ii:%ss","2016.09.21 21:00:00")
b=ParseDate("%yyyy.%mm.%dd %hh:%ii:%ss","2016.09.21 20:50:20")
m=a-n
k=a-b
u.d=k/m
Debug u;=0.000036853028407140316
d.d=u
Debug d;=0.000036853028407140316
g.d=ValD("0.000036853028407140316")
Debug g;=0.000036853028407140316
g.d=0.000036853028407140316
;[19:45:37] [COMPILER] line 12: Numerical overflow: too many digits.
gurj
Enthusiast
Posts: 664 Joined: Thu Jan 22, 2009 3:48 am
Location: china
Contact:
Post
by gurj » Fri Mar 24, 2017 12:12 pm
not bug
should:
Code: Select all
d.d=3.6853028407140316e-5
Debug d;=0.000036853028407140316
gurj
Enthusiast
Posts: 664 Joined: Thu Jan 22, 2009 3:48 am
Location: china
Contact:
Post
by gurj » Fri Mar 24, 2017 12:32 pm
f:
31-30-23-22-00
1 8 23 2^23=8388608
s exp fraction exp=127+指数值
d:
63-62-52-51-00 exp=1023+指数值
1 11 52 2^52=4503599627370496
if (fraction of f.f)>8388608,best use 'e'
if (fraction of d.d)>4503599627370496,best use 'e'
gurj
Enthusiast
Posts: 664 Joined: Thu Jan 22, 2009 3:48 am
Location: china
Contact:
Post
by gurj » Fri Mar 24, 2017 1:18 pm
Code: Select all
;2^52=4503599627370496
;d.d=36853028407140316 best use:
d.d =3685302840714032e1
Debug d
;d.d =0.000036853028407140316 best use:
d.d =3.685302840714032e-5
Debug d
gurj
Enthusiast
Posts: 664 Joined: Thu Jan 22, 2009 3:48 am
Location: china
Contact:
Post
by gurj » Sat Mar 25, 2017 1:09 am
Code: Select all
;***best let ?: 3685302840714032<4503599627370496
;***best use 'e'
d.d =36853028407140320000;***wrong
Debug d;=-40459740278783232.0
d.d =3685302840714032e4;ok
Debug d;=36853028407140319000.0
;ok,but:36853028407140319<>3685302840714032
;so,maybe,*******************
;***for '=', best let:2251799813685248<4503599627370496
; *******************
; 3685302840714032>2251799813685248
; 368530284071403<2251799813685248
; so,3685302840714032 best use:
d.d=368530284071403e1
Debug d;=3685302840714030.0
;368530284071403=368530284071403
;*********
;maybe, f.f for '=', ***best let:4194304<8388608
gurj
Enthusiast
Posts: 664 Joined: Thu Jan 22, 2009 3:48 am
Location: china
Contact:
Post
by gurj » Sat Mar 25, 2017 2:14 am
or best use ValD()
Code: Select all
g.d=ValD("36853028994071450280000000")
Debug g;=36853028994071449000000000.0
g.d=ValD("3.685302899407145028e-8")
Debug g;=0.000000036853028994071449
said
Enthusiast
Posts: 342 Joined: Thu Apr 14, 2011 6:07 pm
Post
by said » Sat Mar 25, 2017 5:39 am
There is certainly a limitation/bug here .... the use of ValD() or scientific form (a.e-n) is just a work around!
gurj
Enthusiast
Posts: 664 Joined: Thu Jan 22, 2009 3:48 am
Location: china
Contact:
Post
by gurj » Sat Mar 25, 2017 6:55 am
maybe has limitation, but no bug:
Code: Select all
;3685302840714032<4503599627370496,so:
d.d=3685302840714032;this is true '='
;but 3685302840714032e4>4503599627370496,so:
d.d =3685302840714032e4;
Debug d;=36853028407140319000.0
;so 36853028407140319<>3685302840714032;
;because:
;3685302840714032e4=64 bit:
;1111111101110000010000100001010010100000111110101000001100000000
;11111111011100000100001000010100101000001111101010000011= 56 bit >52 bit