Page 2 of 3
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Tue Jun 28, 2022 3:29 pm
by Caronte3D
My test was with 1GB too (PB 6 32)
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Tue Jun 28, 2022 6:21 pm
by NicTheQuick
Just write the data in chunks:
Code: Select all
EnableExplicit
; The maximum value is: 2 GiB - 1 Byte = (2 * 1024 * 1024 * 1024 - 1) Bytes = (2^31 - 1) Bytes = 2147483647 Bytes
#MAX_CHUNK_SIZE = 2048 * 1024 * 1024 - 1
Procedure SaveMemory(*Memory, FileName.s, Size = #PB_Ignore)
Protected result.i, file.i
Protected chunkSize.i, writtenBytes.i
file = CreateFile(#PB_Any, FileName)
If Not file
DebuggerError("Could not open file for writing: " + FileName)
ProcedureReturn #False
EndIf
If Size = #PB_Ignore
Size = MemorySize(*Memory)
EndIf
While Size
If Size > #MAX_CHUNK_SIZE
chunkSize = #MAX_CHUNK_SIZE
Else
chunkSize = Size
EndIf
writtenBytes = WriteData(file, *Memory, chunkSize)
Debug writtenBytes
If Not writtenBytes
result = #False
DebuggerError("Could not write " + chunkSize + " bytes into file.")
Break
EndIf
Size - writtenBytes
*Memory + writtenBytes
Wend
CloseFile(file)
ProcedureReturn result
EndProcedure
Define *mem = AllocateMemory(4 * 1024 * 1024 * 1024)
SaveMemory(*mem, "/root/nicolas/tmp/purebasic_memory_test.dat")
FreeMemory(*mem)
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Tue Jun 28, 2022 6:39 pm
by ChrisR
NicTheQuick wrote: Tue Jun 28, 2022 6:21 pm
Just write the data in chunks:
Yes, It worked perfectly now with #MAX_CHUNK_SIZE = 2048 * 1024 * 1024 - 1
With 4Gb allocated
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Tue Jun 28, 2022 6:53 pm
by Caronte3D
NicTheQuick wrote: Tue Jun 28, 2022 6:21 pm
Just write the data in chunks:
Candidate to Tricks 'n' Tips forum?
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Wed Jun 29, 2022 12:14 am
by Simo_na
NicTheQuick wrote: Tue Jun 28, 2022 6:21 pm
Just write the data in chunks:
Yes, try to change the chunks size and monitoring the disk efficiency with this app:
https://www.sysgauge.com
you will see the results changing according to the block size
But with 'SaveMemory' command ...is not more comfortable ?
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Wed Jun 29, 2022 7:10 am
by Bisonte
However, I now see that WriteData() has a 2GB limit.
It looks like the function works internally with longs (so a 32 bit limit).
That is maybe a bug report for the x64 version of pb ?
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Wed Jun 29, 2022 9:43 am
by NicTheQuick
Bisonte wrote: Wed Jun 29, 2022 7:10 am
However, I now see that WriteData() has a 2GB limit.
It looks like the function works internally with longs (so a 32 bit limit).
That is maybe a bug report for the x64 version of pb ?
I don't think it can be changed easily because the underlying functions by itself uses 32 bit parameters. See
fwrite which uses `size_t` which in turn usually is `unsigned int` and `int` usually only has 32 bits.
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Wed Jun 29, 2022 11:16 am
by Simo_na
NicTheQuick wrote: Wed Jun 29, 2022 9:43 am
I don't think it can be changed easily because the underlying functions by itself uses 32 bit parameters. See
it's a 'bit' limiting/obsolete for today's times ...
can be do with the Assembler ???
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Wed Jun 29, 2022 11:28 am
by NicTheQuick
Simo_na wrote: Wed Jun 29, 2022 11:16 am
it's a 'bit' limiting/obsolete for today's times ...
can be do with the Assembler ???
But why? It's not limiting. It's quite unusual to write such big memory chunks into a file at once. Just split it up in small chunks and all is good.
And no, Assembler is not a solution for everything. You must use the functions the kernel offers you to write into files and other things. They can not be replaced with your own user space functions.
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Wed Jun 29, 2022 11:56 am
by Simo_na
NicTheQuick wrote: Wed Jun 29, 2022 11:28 am
But why? It's not limiting. It's quite unusual to write such big memory chunks into a file at once. Just split it up in small chunks and all is good.
Thank you
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Wed Jun 29, 2022 12:22 pm
by jacdelad
Simo_na wrote: Wed Jun 29, 2022 11:16 am
NicTheQuick wrote: Wed Jun 29, 2022 9:43 am
I don't think it can be changed easily because the underlying functions by itself uses 32 bit parameters. See
it's a 'bit' limiting/obsolete for today's times ...
can be do with the Assembler ???
No, no, not assembler. This will not work with the C compiler (at least for now). Also I don't see a problem to chunk it. The result is the same.
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Wed Jun 29, 2022 4:08 pm
by Simo_na
did some test with chunk dimension...
the best (fastest in write) size is 2097120 for my system.
Thank you
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Wed Jun 29, 2022 4:30 pm
by jacdelad
That's suspiciously close to two 2MB and somehow I doubt it, because this way the loop is run 1000 times for 2 GB, instead of once.
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Wed Jun 29, 2022 5:32 pm
by NicTheQuick
jacdelad wrote: Wed Jun 29, 2022 4:30 pm
That's suspiciously close to two 2MB and somehow I doubt it, because this way the loop is run 1000 times for 2 GB, instead of once.
Keep in mind that your HDD/SSD is way slower than your program code. Also in the background data gets not transferred to disk in the same moment you call WriteData(). Instead it goes into a memory cache and gets flushed to disk when your kernel thinks it is the time to do so.
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Thu Jun 30, 2022 4:42 am
by jacdelad
@Nic: Exactly, that sounds like another argument to make the chunks as big as possible, because the data transfer to the cache is faster.