Size limit of CatchXML?

[Lebostein]

I have a very big XML file (1 GB without spaces). There are some encoded thumbnails inside (that is the reason for the size). With Python I can open this thing (with the streaming option). With PureBasic I get the PB intern "out of memory" error while parsing. But I have about 7 GB memory free...

[miso]

Also theres an image (not sprite) size limit. (though did not test this recently)

[Lebostein]

Also theres an image (not sprite) size limit.

I haven't gotten to the point of reading the images yet. I have loaded the file into memory and would like to parse the XML with PB.

Code: Select all

filename$ = "/Users/lebostein/lib.xml"
fileid = ReadFile(#PB_Any, filename$)
*memory = AllocateMemory(Lof(fileid) - Loc(fileid))
ReadData(fileid, *memory, MemorySize(*memory))
CloseFile(fileid)
ID = CatchXML(#PB_Any, *memory, MemorySize(*memory))
Debug XMLError(ID) ; --> "out of memory"

If I load directly, it seems to work:

Code: Select all

filename$ = "/Users/lebostein/lib.xml"
ID = LoadXML(#PB_Any, filename$)
Debug XMLError(ID) ; --> ""

The first version only requires 1 GB more RAM. But actually, there is enough free RAM, even with the 1 GB reservation. I have 16 GB. 5 are used by system an other processes

[Lebostein]

Very strange. If I load the 1 GB XML file in the memory and catch the XML I get the "out of memory" error. When I load directly from the hard drive it works. But if I load the file into the memory first and I still read from the disk afterwards (without releasing the occupied memory), it works again. Either there is something wrong with CatchXML, or CatchXML is extremely memory-intensive (should use around 7 GB of memory, because that's how much is still free shortly before)....

Code: Select all

filename$ = "/Users/lebostein/x.xml" ; 1 GB XML file, formated without spaces and without line breaks

Select 1

  Case 1
  
    ; Load file into memory
    fileid = ReadFile(#PB_Any, filename$)
    *memory = AllocateMemory(Lof(fileid))
    ReadData(fileid, *memory, MemorySize(*memory))
    CloseFile(fileid)

    ; catch from memory
    XML = CatchXML(#PB_Any, *memory, MemorySize(*memory))
    If XMLStatus(XML): Debug XMLError(XML): Else: Debug "OK": EndIf ; ---> "out of memory"
  
  Case 2

    ; Load XML from disc
    XML = LoadXML(#PB_Any, filename$)
    If XMLStatus(XML): Debug XMLError(XML): Else: Debug "OK": EndIf ; ---> OK

  Case 3

    ; Load file into memory
    fileid = ReadFile(#PB_Any, filename$)
    *memory = AllocateMemory(Lof(fileid))
    ReadData(fileid, *memory, MemorySize(*memory))
    CloseFile(fileid)

    ; Load XML from disc
    XML = LoadXML(#PB_Any, filename$)
    If XMLStatus(XML): Debug XMLError(XML): Else: Debug "OK": EndIf ; ---> OK

EndSelect

[pfnuesu]

Just to eliminate the obvious things: You're using the x64 compiler, right?

[Lebostein]

Just to eliminate the obvious things: You're using the x64 compiler, right?

Sorry, I use MacOS 15.7.3 (16 GB RAM, MacBook Air M3). There is no 32-Bit version. I tested both:
PureBasic 6.30 b6 C Backend (x64)
PureBasic 6.30 b6 C Backend (arm64)