Just starting out? Need help? Post your questions and find answers here.
sverson
Enthusiast
Posts: 286 Joined: Sun Jul 04, 2004 12:15 pm
Location: Germany
Post
by sverson » Sat Oct 16, 2010 12:52 am
Regarding: '
ReadData (#File, *MemoryBuffer,
LengthToRead )'
Is there a 16MB ($1000000) limit at '
LengthToRead '?
Example:
Code: Select all
MaxBufferSize = $1000000
StandardFile$ = ""
Pattern$ = "All files (*.*)|*.*"
Pattern = 0
File$ = OpenFileRequester("Please choose file to load", StandardFile$, Pattern$, Pattern)
Debug "FileName: " + File$
If File$
If ReadFile(0, File$)
length.q = Lof(0)
Debug "FileSize: " + Str(length)
If length > MaxBufferSize
BufferSize = MaxBufferSize
Else
BufferSize = length
EndIf
Debug "bufferSize: " + Str(BufferSize)
*MemoryID = AllocateMemory(BufferSize)
If *MemoryID
bytes = ReadData(0, *MemoryID, BufferSize)
Debug "Number of bytes read: " + Str(bytes)
FreeMemory(*MemoryID)
EndIf
CloseFile(0)
Else
Debug "Couldn't open the file!"
EndIf
Else
Debug "The requester was canceled."
EndIf
This works fine up to
MaxBufferSize = $1000000 .
FileName: C:\verybigfile.txt
FileSize: 7256495952
bufferSize: 16777216
Number of bytes read: 16777216
...but fails at
MaxBufferSize >= $1000001 .
FileName: C:\verybigfile.txt
FileSize: 7256495952
bufferSize: 16777217
Number of bytes read: 0
I need to handle very big files as fast as possible. Is there a workaround?
Last edited by
sverson on Sat Oct 16, 2010 10:36 am, edited 1 time in total.
netmaestro
PureBasic Bullfrog
Posts: 8451 Joined: Wed Jul 06, 2005 5:42 am
Location: Fort Nelson, BC, Canada
Post
by netmaestro » Sat Oct 16, 2010 2:13 am
Reading in chunks of say 4096 bytes with the last one being smaller is a good solution for streaming it in. Imho it would be just as fast if not faster than reading the whole block in at once because the Purebasic file library is going to stream it anyway iirc.
BERESHEIT
sverson
Enthusiast
Posts: 286 Joined: Sun Jul 04, 2004 12:15 pm
Location: Germany
Post
by sverson » Sat Oct 16, 2010 2:35 am
Thanks netmaestro,
I will try this way and compare the speed.
PB
PureBasic Expert
Posts: 7581 Joined: Fri Apr 25, 2003 5:24 pm
Post
by PB » Sat Oct 16, 2010 2:58 am
> I need to handle very big files as fast as possible
Your code works just fine here with a 7.85 GB file. Windows XP, 32bit.
I compile using 5.31 (x86) on Win 7 Ultimate (64-bit).
"PureBasic won't be object oriented, period" - Fred.
sverson
Enthusiast
Posts: 286 Joined: Sun Jul 04, 2004 12:15 pm
Location: Germany
Post
by sverson » Sat Oct 16, 2010 8:58 am
PB wrote: Your code works just fine here with a 7.85 GB file. Windows XP, 32bit.
Set MaxBufferSize to $1000001 or bigger.
PB
PureBasic Expert
Posts: 7581 Joined: Fri Apr 25, 2003 5:24 pm
Post
by PB » Sat Oct 16, 2010 9:44 am
> Set MaxBufferSize to $1000001 or bigger
Still no problem, even with a MaxBufferSize set to $5000000:
I compile using 5.31 (x86) on Win 7 Ultimate (64-bit).
"PureBasic won't be object oriented, period" - Fred.
sverson
Enthusiast
Posts: 286 Joined: Sun Jul 04, 2004 12:15 pm
Location: Germany
Post
by sverson » Sat Oct 16, 2010 10:35 am
This seems to be the 1st severe VirtualBox issue I discovered since I'm running all PB environments inside VirtualBoxes.
The code works fine on the host system but not in the box.