PureBASIC limited memory allocation?

[SnowyDog]

Hi all

I am a seasoned PowerBASIC user but am interested in PureBASIC, mainly because I will soon need to port some of my existing PowerBASIC applications to Linux.

My applications typically involve processing large disk files (often as large as 1TiB). To optimise speed (disk accesses being the most significant bottleneck) my normal approach is to allocate as much memory as possible and then load chunks of the file into memory for processing.

Using PowerBASIC I am able to comfortably allocate a 1GiB (1073741824 byte) buffer on my Windows XP 32-bit box. However using PureBASIC (I am currently using the demo version of the Windows 32-bit compiler) the memory allocation fails. It does allow me to allocate 500MiB (536870912 bytes) however.

Here is the PureBASIC code:-

Code: Select all

EnableExplicit
Define FileName.s, FileHandle.l, FileSize.q, BytesRead.q, *MemoryID

FileName = OpenFileRequester("Select a file","","Binary (.bin)|*.bin|All files (*.*)|*.*",0)
If FileName<>""
  FileHandle=ReadFile(#PB_Any,FileName)
  If FileHandle
    FileSize=Lof(FileHandle)
    Debug "File size: "+Str(FileSize)+" bytes."
    *MemoryID=AllocateMemory(FileSize,#PB_Memory_NoClear)
    If *MemoryID
      Debug "Allocated: "+Str(MemorySize(*MemoryID))+" bytes."
      BytesRead = ReadData(FileHandle, *MemoryID, FileSize)
      Debug "Number of bytes read: " + Str(BytesRead)
    Else
      Debug "Failed to allocate memory."
    EndIf
    CloseFile(FileHandle)
    If *MemoryID
      FreeMemory(*MemoryID)
      Debug "Memory freed."
    EndIf
  Else
    Debug "Could not open or read from selected file."
  EndIf
Else
  Debug "No file selected."
EndIf

Does anyone know the reason why I am unable to allocate 1GiB of memory using PureBASIC but yet on the same box I am able to allocate 1GiB using my equivalent PowerBASIC code?

Thanks
Robin

[Fred]

Seems to work here on PB x86:

Code: Select all

Debug AllocateMemory(1073741824)

PB uses standard HeapAlloc() function, so there is no reason it would fail. You can also use the x64 version to have more available memory.

BTW, 1GB buffer is overkill for disk operation, 100 MB or less should be more than enough to avoid small read penalities. Also PB has built-in buffered read for files, checkout the FileBuffersSize() http://www.purebasic.com/documentation/ ... ssize.html command.

[rsts]

Worked fine with 1.2GB file

windows s64 PBx86

[coco2]

I've seen stock trading analysis software that works with very large files and it performs MUCH faster with only the difference being the size of the cache on the hard disk itself (if I remember the HDD with 8mb took maybe 5 minues yet the HDD with 32mb cache took only 30 seconds)

[davido]

Might the problem be due to the restrictions placed upon the Demo Version?

[Fred]

Might the problem be due to the restrictions placed upon the Demo Version?

No, it should not be any difference regarding this topic.

[spacebuddy]

I just tried it with a 2G file and it loaded in 2 seconds.

file$ = OpenFileRequester("Select a file","","Text (.txt)|*.txt|All files (*.*)|*.*",0)
If file$
If ReadFile(0, file$)
length = Lof(0) ; get the length of opened file
*MemoryID = AllocateMemory(length) ; allocate the needed memory
If *MemoryID
bytes = ReadData(0, *MemoryID, length) ; read all data into the memory block
Debug "Number of bytes read: " + Str(bytes)
EndIf
CloseFile(0)
EndIf
EndIf