Page 3 of 3
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Thu Jun 30, 2022 3:22 pm
by Simo_na
Code: Select all
EnableExplicit
DisableDebugger
Global.q gigs = 4*1024*1024*1024,mytime1,mytime
Procedure Netmestro(mysize)
Protected *MemoryID,*writeptr,*end
Protected.i chunksize
*MemoryID = AllocateMemory(gigs)
If *MemoryID > 0
FillMemory(*MemoryID,gigs,255,#PB_Byte)
If CreateFile(0,GetCurrentDirectory()+"test.tmp")
;FileBuffersSize(0,mysize)
*writeptr=*MemoryID
*end = *MemoryID + gigs
While *writeptr < *end
If *end-*writeptr >= mysize
chunksize = mysize
Else
chunksize = *end-*writeptr
EndIf
WriteData(0, *writeptr, chunksize)
*writeptr + chunksize
Wend
;FlushFileBuffers(0)
CloseFile(0)
EndIf
EndIf
EndProcedure
mytime1=ElapsedMilliseconds()
Netmestro(67108864)
mytime=ElapsedMilliseconds()-mytime1
MessageRequester("mytime",Str(mytime))
; Chunck 512 = 4014 Elapsed time
; Chunck 1024 = 4627 Elapsed time
; Chunck 2048 = 5048 Elapsed time
; Chunck 4096 = 4238 Elapsed time
; Chunck 8192 = 2965 Elapsed time
; Chunck 16384 = 2287 Elapsed time
; Chunck 32768 = 1864 Elapsed time
; Chunck 65536 = 1596 Elapsed time
; Chunck 131072 = 1660 Elapsed time
; Chunck 262144 = 1570 Elapsed time
; Chunck 524288 = 1468 Elapsed time
; Chunck 1048576 = 1499 Elapsed time
; Chunck 2097152 = 1569 Elapsed time
; Chunck 4194304 = 1576 Elapsed time
; Chunck 8388608 = 1468 Elapsed time
; Chunck 16777216 = 1468 Elapsed time
; Chunck 33554432 = 2143 Elapsed time
; Chunck 67108864 = 2529 Elapsed time
if you really want to be precise you can add this...but it is not necessary.
Code: Select all
Repeat
mydimension=FileSize(GetCurrentDirectory()+"test.tmp")
Until mydimension = 4294967296
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Thu Jun 30, 2022 4:08 pm
by NicTheQuick
You are also measuring creating and filling the memory with your test. Was that intentional?
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Thu Jun 30, 2022 4:28 pm
by NicTheQuick
However I used your code and tested it against my SSD, NVME and tmpfs (RAM) and I am impressed by your speeds. How do you write 2,7 GB/s to your Harddrive? Are you using a NVME RAID or is it a NVME PCIe 4.0 drive?
Code: Select all
; User Simo_na NicTheQuick
; SSD/RAM/NVME SSD RAM NVME
; Chunk 512 = 4014 7692 2721 3222
; Chunk 1024 = 4627 6546 2895 4026
; Chunk 2048 = 5048 10497 2776 3602
; Chunk 4096 = 4238 7744 2546 3210
; Chunk 8192 = 2965 10761 2541 3485
; Chunk 16384 = 2287 8322 2330 3297
; Chunk 32768 = 1864 9978 2255 3288
; Chunk 65536 = 1596 7303 2140 3120
; Chunk 131072 = 1660 10696 2200 3144
; Chunk 262144 = 1570 9361 2068 3057
; Chunk 524288 = 1468 6517 2083 3121
; Chunk 1048576 = 1499 8969 2029 3161
; Chunk 2097152 = 1569 6944 2035 3120
; Chunk 4194304 = 1576 9490 2043 3105
; Chunk 8388608 = 1468 7371 2019 3108
; Chunk 16777216 = 1468 9304 2076 3085
; Chunk 33554432 = 2143 9997 2020 3109
; Chunk 67108864 = 2529 7450 2037 3115
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Thu Jun 30, 2022 4:52 pm
by NicTheQuick
I changed the code so AllocateMemory() gets no longer measured. And now the image is much clearer.
This is the test code:
Code: Select all
EnableExplicit
; The maximum value is: 2 GiB - 1 Byte = (2 * 1024 * 1024 * 1024 - 1) Bytes = (2^31 - 1) Bytes = 2147483647 Bytes
Global MAX_CHUNK_SIZE = 2048 * 1024 * 1024 - 1
Procedure SaveMemory(*Memory, FileName.s, Size = #PB_Ignore)
Protected result.i, file.i
Protected chunkSize.i, writtenBytes.i
file = CreateFile(#PB_Any, FileName)
If Not file
DebuggerError("Could not open file for writing: " + FileName)
ProcedureReturn #False
EndIf
If Size = #PB_Ignore
Size = MemorySize(*Memory)
EndIf
While Size
If Size > MAX_CHUNK_SIZE
chunkSize = MAX_CHUNK_SIZE
Else
chunkSize = Size
EndIf
writtenBytes = WriteData(file, *Memory, chunkSize)
If Not writtenBytes
result = #False
DebuggerError("Could not write " + chunkSize + " bytes into file.")
Break
EndIf
Size - writtenBytes
*Memory + writtenBytes
Wend
CloseFile(file)
ProcedureReturn result
EndProcedure
OpenConsole()
Define *mem = AllocateMemory(4 * 1024 * 1024 * 1024)
Define time.i, chunkSizeBits.i, chunkSize.i, file.s, result.s
file.s = "/test/purebasic_memory_test.dat"
PrintN("Chunk Size Time")
For chunkSizeBits = 9 To 30
MAX_CHUNK_SIZE = 1 << chunkSizeBits
DeleteFile(file)
Delay(100)
time = ElapsedMilliseconds()
SaveMemory(*mem, file)
time = ElapsedMilliseconds() - time
PrintN(RSet(Str(MAX_CHUNK_SIZE), 10, " ") + " " + RSet(Str(time), 5, " "))
Next
FreeMemory(*mem)
Input()
CloseConsole()
And now the results for NVME, RAM and SSD, each writing a 4 GiB file.
Code: Select all
; NVME
Chunk Size Time
512 2133
1024 1831
2048 1626
4096 1534
8192 1564
16384 1453
32768 1330
65536 1415
131072 1294
262144 1620
524288 1254
1048576 1258
2097152 1512
4194304 1427
8388608 1464
16777216 1453
33554432 1421
67108864 1507
134217728 1401
268435456 1421
536870912 1493
1073741824 1352
Code: Select all
; RAM
Chunk Size Time
512 1738
1024 1485
2048 1244
4096 1121
8192 1124
16384 888
32768 770
65536 705
131072 674
262144 653
524288 660
1048576 648
2097152 647
4194304 646
8388608 643
16777216 646
33554432 652
67108864 646
134217728 652
268435456 649
536870912 644
1073741824 651
Code: Select all
; SSD
Chunk Size Time
512 5763
1024 5353
2048 6672
4096 4438
8192 3536
16384 4222
32768 3501
65536 4736
131072 7188
262144 4606
524288 6663
1048576 5113
2097152 5132
4194304 5569
8388608 5884
16777216 7913
33554432 5448
67108864 7863
134217728 5739
268435456 8021
536870912 4836
1073741824 5004
As you can see, the NVME gets not much faster after a chunk size of 16 kiB, RAM does not increase anymore after a chunk size of 128 kiB and the SSD is just all over the place.
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Thu Jun 30, 2022 6:06 pm
by Simo_na
Thank you NicTheQuick, nice !!
This is my NVME (WD SN850 - 500GB @ half full...203GB free)
Code: Select all
NVME
Chunk Size Time Nic Chunk Size Time Simo
512 2133 512 4251
1024 1831 1024 4108
2048 1626 2048 4829
4096 1534 4096 3722
8192 1564 8192 2417
16384 1453 16384 1500
32768 1330 32768 1164
65536 1415 65536 922
131072 1294 131072 820
262144 1620 262144 780
524288 1254 524288 775
1048576 1258 1048576 750
2097152 1512 2097152 716
4194304 1427 4194304 700
8388608 1464 8388608 697
16777216 1453 16777216 686
33554432 1421 33554432 1431
67108864 1507 67108864 1906
134217728 1401 134217728 1993
268435456 1421 268435456 2259
536870912 1493 536870912 2146
1073741824 1352 1073741824 2573
the 33554432 and 16384 is similar ...
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Thu Jun 30, 2022 6:29 pm
by Simo_na
With imdisk (
)
https://sourceforge.net/projects/imdisk-toolkit/
Code: Select all
Chunk Size Time
512 12185
1024 13923
2048 20464
4096 70
8192 48
16384 37
32768 29
65536 24
131072 20
262144 18
524288 16
1048576 16
2097152 15
4194304 15
8388608 15
16777216 14
33554432 18
67108864 19
134217728 0
268435456 0
536870912 0
1073741824 0
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Thu Jun 30, 2022 6:39 pm
by NicTheQuick
These values make no sense at all
Re: Feature Request 'SaveMemory' to Disk Drive
Posted: Thu Jun 30, 2022 9:58 pm
by Simo_na
Sure, I was joking..
Thanks for the help.