Page 1 of 1

ReadData: LengthToRead limit? [Resolved]

Posted: Sat Oct 16, 2010 12:52 am
by sverson
Regarding: 'ReadData(#File, *MemoryBuffer, LengthToRead)'
Is there a 16MB ($1000000) limit at 'LengthToRead'?

Example:

Code: Select all

MaxBufferSize = $1000000
StandardFile$ = ""
Pattern$ = "All files (*.*)|*.*"
Pattern = 0
File$ = OpenFileRequester("Please choose file to load", StandardFile$, Pattern$, Pattern)
Debug "FileName: " + File$
If File$
  If ReadFile(0, File$)
    length.q = Lof(0)
    Debug "FileSize: " + Str(length)
    If length > MaxBufferSize
      BufferSize = MaxBufferSize
    Else
      BufferSize = length
    EndIf
    Debug "bufferSize: " + Str(BufferSize)
    *MemoryID = AllocateMemory(BufferSize)
    If *MemoryID
      bytes = ReadData(0, *MemoryID, BufferSize)
      Debug "Number of bytes read: " + Str(bytes)
      FreeMemory(*MemoryID)
    EndIf
    CloseFile(0)
  Else
    Debug "Couldn't open the file!"
  EndIf
Else
  Debug "The requester was canceled."
EndIf
This works fine up to MaxBufferSize = $1000000.
FileName: C:\verybigfile.txt
FileSize: 7256495952
bufferSize: 16777216
Number of bytes read: 16777216
...but fails at MaxBufferSize >= $1000001.
FileName: C:\verybigfile.txt
FileSize: 7256495952
bufferSize: 16777217
Number of bytes read: 0
I need to handle very big files as fast as possible. Is there a workaround?

Re: ReadData: LengthToRead limit?

Posted: Sat Oct 16, 2010 2:13 am
by netmaestro
Reading in chunks of say 4096 bytes with the last one being smaller is a good solution for streaming it in. Imho it would be just as fast if not faster than reading the whole block in at once because the Purebasic file library is going to stream it anyway iirc.

Re: ReadData: LengthToRead limit?

Posted: Sat Oct 16, 2010 2:35 am
by sverson
Thanks netmaestro,
I will try this way and compare the speed.

Re: ReadData: LengthToRead limit?

Posted: Sat Oct 16, 2010 2:58 am
by PB
> I need to handle very big files as fast as possible

Your code works just fine here with a 7.85 GB file. Windows XP, 32bit.

Image

Re: ReadData: LengthToRead limit?

Posted: Sat Oct 16, 2010 8:58 am
by sverson
PB wrote:Your code works just fine here with a 7.85 GB file. Windows XP, 32bit.
Set MaxBufferSize to $1000001 or bigger. :wink:

Re: ReadData: LengthToRead limit?

Posted: Sat Oct 16, 2010 9:44 am
by PB
> Set MaxBufferSize to $1000001 or bigger

Still no problem, even with a MaxBufferSize set to $5000000:

Image

Re: ReadData: LengthToRead limit?

Posted: Sat Oct 16, 2010 10:35 am
by sverson
This seems to be the 1st severe VirtualBox issue I discovered since I'm running all PB environments inside VirtualBoxes.

The code works fine on the host system but not in the box.