ReadData: LengthToRead limit? [Resolved]

[sverson]

Regarding: 'ReadData(#File, *MemoryBuffer, LengthToRead)'
Is there a 16MB ($1000000) limit at 'LengthToRead'?

Example:

Code: Select all

MaxBufferSize = $1000000
StandardFile$ = ""
Pattern$ = "All files (*.*)|*.*"
Pattern = 0
File$ = OpenFileRequester("Please choose file to load", StandardFile$, Pattern$, Pattern)
Debug "FileName: " + File$
If File$
  If ReadFile(0, File$)
    length.q = Lof(0)
    Debug "FileSize: " + Str(length)
    If length > MaxBufferSize
      BufferSize = MaxBufferSize
    Else
      BufferSize = length
    EndIf
    Debug "bufferSize: " + Str(BufferSize)
    *MemoryID = AllocateMemory(BufferSize)
    If *MemoryID
      bytes = ReadData(0, *MemoryID, BufferSize)
      Debug "Number of bytes read: " + Str(bytes)
      FreeMemory(*MemoryID)
    EndIf
    CloseFile(0)
  Else
    Debug "Couldn't open the file!"
  EndIf
Else
  Debug "The requester was canceled."
EndIf

This works fine up to MaxBufferSize = $1000000.

FileName: C:\verybigfile.txt
FileSize: 7256495952
bufferSize: 16777216
Number of bytes read: 16777216

...but fails at MaxBufferSize >= $1000001.

FileName: C:\verybigfile.txt
FileSize: 7256495952
bufferSize: 16777217
Number of bytes read: 0

I need to handle very big files as fast as possible. Is there a workaround?

[netmaestro]

Reading in chunks of say 4096 bytes with the last one being smaller is a good solution for streaming it in. Imho it would be just as fast if not faster than reading the whole block in at once because the Purebasic file library is going to stream it anyway iirc.

[sverson]

Thanks netmaestro,
I will try this way and compare the speed.

[PB]

> I need to handle very big files as fast as possible

Your code works just fine here with a 7.85 GB file. Windows XP, 32bit.

[sverson]

Your code works just fine here with a 7.85 GB file. Windows XP, 32bit.

Set MaxBufferSize to $1000001 or bigger.

[PB]

> Set MaxBufferSize to $1000001 or bigger

Still no problem, even with a MaxBufferSize set to $5000000:

[sverson]

This seems to be the 1st severe VirtualBox issue I discovered since I'm running all PB environments inside VirtualBoxes.

The code works fine on the host system but not in the box.