Reading huge files
Posted: Tue Jun 23, 2020 5:30 pm
Hi. all! There is such a task - to read very large files (up to 10 GB) for further processing.
An array element is created for each line of the file.
After reading the forums I got such a fragment :
The problem is that if the file size grows somewhere around 242 or higher, the program stops with an error
[14:42:52] [ERROR] Invalid memory access. (write error at address 0)
The error is mainly in the line: String=PeekS(pointMemForReadFile,MemorySize(pointMemForReadFile),#PB_UTF8)
Sometimes in line : StringArray(i) = S\s
The amount of available memory was calculated during operation using MemoryStatus (#PB_System_FreePhysical) and it is about 10GB
I have no experience and good knowledge of working with memory, just the usual line-by-line reading of files is absolutely not suitable
by speed. Need constructive advice on how to best solve the problem.
An array element is created for each line of the file.
After reading the forums I got such a fragment :
Code: Select all
Procedure ReadFileIntoArray(file$, Array StringArray.s(1), Separator.s = " ")
Define String.s ; The line to which we copy the memory area containing the entire read file
Protected S.String, *S.Integer = @S
Protected.i countFileString, i, pos_separ, slen
file_handler=ReadFile(#PB_Any,file$)
If file_handler
ReadStringFormat(file_handler)
lengthFile.q=(Lof(file_handler)-Loc(file_handler)) -2
If lengthFile>0
pointMemForReadFile=AllocateMemory(lengthFile)
If pointMemForReadFile
numberBytesReadingFromFile = ReadData(file_handler,pointMemForReadFile,lengthFile)
String=PeekS(pointMemForReadFile,MemorySize(pointMemForReadFile),#PB_UTF8)
countFileString = CountString(String, Separator)
slen = Len(Separator)
ReDim StringArray(countFileString)
*S\i = @String
While i < countFileString
pos_separ = FindString(S\s, Separator) separator "Separator"
StringArray(i) = PeekS(*S\i, pos_separ - 1)
*S\i + (pos_separ + slen - 1) << #PB_Compiler_Unicode
i + 1
Wend
StringArray(i) = S\s
*S\i = 0
FreeMemory(pointMemForReadFile)
String=""
Else
Debug "Memory Allocation error"
EndIf
EndIf
CloseFile(file_handler)
EndIf
EndProcedure
Dim LogString$(0); An array in which each line of the file will be read
ReadFileIntoArray("e:\0YP\Purebasic\LogAnalyzer\Log\test.log", LogString$() , Chr(10))
CountLogString=ArraySize(LogString$()) ; The number of lines in the read file
Debug LogString$(0) ; Print the first line of the file
Debug LogString$(CountLogString) ; Print the last line of the file
[14:42:52] [ERROR] Invalid memory access. (write error at address 0)
The error is mainly in the line: String=PeekS(pointMemForReadFile,MemorySize(pointMemForReadFile),#PB_UTF8)
Sometimes in line : StringArray(i) = S\s
The amount of available memory was calculated during operation using MemoryStatus (#PB_System_FreePhysical) and it is about 10GB
I have no experience and good knowledge of working with memory, just the usual line-by-line reading of files is absolutely not suitable
by speed. Need constructive advice on how to best solve the problem.