Page 1 of 1
WriteData() 2GB size limit
Posted: Sun Jul 07, 2024 1:48 am
by technopol
According to help, WriteData() still support only up to 2GB.
Is there a way to write more at once, even with #PB_File_NoBuffering set?
Re: WriteData() 2GB size limit
Posted: Sun Jul 07, 2024 2:18 am
by BarryG
For/Next with each next 2 GB block, along with FileSeek?
Re: WriteData() 2GB size limit
Posted: Sun Jul 07, 2024 3:21 am
by technopol
BarryG wrote: Sun Jul 07, 2024 2:18 am
For/Next with each next 2 GB block, along with FileSeek?
Thanks for the suggestion.
Of course I can do it in 2GB blocks, in sequence, no need for FileSeek for that. What I want is read a file, let's say 20GB, in memory, work on it, then save it in one WriteData. But if it's impossible, I'll go with 2GB writing and buffering. The time I will lose by releasing my disk operating system priority to anything else shall be minimal with this block size anyway. Also, my work buffer shall be much smaller.
It's just that, since Purebasic 6.11 is mostly 64-bit compatible now, I was hoping the 2GB WriteData() limit was only a forgotten help file update.
Re: WriteData() 2GB size limit
Posted: Sun Jul 07, 2024 2:54 pm
by SMaag
under Windows you can use FileMapping for large files. But don't aske me how to do?
https://learn.microsoft.com/en-us/windo ... le-mapping
File mapping is the association of a file's contents with a portion of the virtual address space of a process. The system creates a file mapping object (also known as a section object) to maintain this association. A file view is the portion of virtual address space that a process uses to access the file's contents. File mapping allows the process to use both random input and output (I/O) and sequential I/O. It also allows the process to work efficiently with a large data file, such as a database, without having to map the whole file into memory. Multiple processes can also use memory-mapped files to share data.
Re: WriteData() 2GB size limit
Posted: Sun Jul 07, 2024 4:36 pm
by NicTheQuick
The limitation comes from the API itself, not from Purebasic, I think.
You could probably use mmap to work directly with pointers:
Code: Select all
EnableExplicit
Define hFile.i
hFile = OpenFile(#PB_Any, "/home/nicolas/update.txt")
If Not hFile
DebuggerWarning("Could not open file.")
End
EndIf
Define size.i = Lof(hFile)
Debug "Size: " + size
Enumeration
#PROT_READ = $01
#PROT_WRITE = $02
#MAP_SHARED = $01
EndEnumeration
Define *ptr = mmap64_(#Null, size, #PROT_READ | #PROT_WRITE, #MAP_SHARED, fileno_(FileID(hFile)), 0)
If *ptr = -1
DebuggerWarning("Could not map file.")
CloseFile(hFile)
End
EndIf
; Inhalt über Pointer ausgeben
Debug PeekS(*ptr, -1, #PB_UTF8)
munmap_(*ptr, size)
CloseFile(hFile)
But if you want to write to the file you probably need to use ftruncate_() to increase the size of the file before mapping it to memory and writing in it.
Oh, and this is only tested on Linux. I don't know about Windows.
Re: WriteData() 2GB size limit
Posted: Sun Jul 07, 2024 7:14 pm
by technopol
Interesting solutions, but since I need to keep the program easily portable (Win, Linux, Mac and eventually Android and iOS) as it will be sold as libraries in a SDK, I will keep it to 2GB for now.