WriteData() 2GB size limit

[technopol]

According to help, WriteData() still support only up to 2GB.
Is there a way to write more at once, even with #PB_File_NoBuffering set?

[BarryG]

For/Next with each next 2 GB block, along with FileSeek?

[technopol]

For/Next with each next 2 GB block, along with FileSeek?

Thanks for the suggestion.

Of course I can do it in 2GB blocks, in sequence, no need for FileSeek for that. What I want is read a file, let's say 20GB, in memory, work on it, then save it in one WriteData. But if it's impossible, I'll go with 2GB writing and buffering. The time I will lose by releasing my disk operating system priority to anything else shall be minimal with this block size anyway. Also, my work buffer shall be much smaller.

It's just that, since Purebasic 6.11 is mostly 64-bit compatible now, I was hoping the 2GB WriteData() limit was only a forgotten help file update.

[SMaag]

under Windows you can use FileMapping for large files. But don't aske me how to do?

https://learn.microsoft.com/en-us/windo ... le-mapping

File mapping is the association of a file's contents with a portion of the virtual address space of a process. The system creates a file mapping object (also known as a section object) to maintain this association. A file view is the portion of virtual address space that a process uses to access the file's contents. File mapping allows the process to use both random input and output (I/O) and sequential I/O. It also allows the process to work efficiently with a large data file, such as a database, without having to map the whole file into memory. Multiple processes can also use memory-mapped files to share data.

[NicTheQuick]

The limitation comes from the API itself, not from Purebasic, I think.

You could probably use mmap to work directly with pointers:

Code: Select all

EnableExplicit
Define hFile.i

hFile = OpenFile(#PB_Any, "/home/nicolas/update.txt")
If Not hFile
	DebuggerWarning("Could not open file.")
	End
EndIf

Define size.i = Lof(hFile)

Debug "Size: " + size

Enumeration
	#PROT_READ = $01
	#PROT_WRITE = $02
	#MAP_SHARED = $01
EndEnumeration

Define *ptr = mmap64_(#Null, size, #PROT_READ | #PROT_WRITE, #MAP_SHARED, fileno_(FileID(hFile)), 0)
If *ptr = -1
	DebuggerWarning("Could not map file.")
	CloseFile(hFile)
	End
EndIf

; Inhalt über Pointer ausgeben
Debug PeekS(*ptr, -1, #PB_UTF8)

munmap_(*ptr, size)
CloseFile(hFile)

But if you want to write to the file you probably need to use ftruncate_() to increase the size of the file before mapping it to memory and writing in it.

Oh, and this is only tested on Linux. I don't know about Windows.

[technopol]

Interesting solutions, but since I need to keep the program easily portable (Win, Linux, Mac and eventually Android and iOS) as it will be sold as libraries in a SDK, I will keep it to 2GB for now.