Feature Request 'SaveMemory' to Disk Drive

Got an idea for enhancing PureBasic? New command(s) you'd like to see?
Simo_na
Enthusiast
Enthusiast
Posts: 177
Joined: Sun Mar 03, 2013 9:01 am

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Simo_na »

Code: Select all

EnableExplicit
DisableDebugger

Global.q gigs = 4*1024*1024*1024,mytime1,mytime

Procedure Netmestro(mysize)
Protected *MemoryID,*writeptr,*end
Protected.i chunksize

*MemoryID = AllocateMemory(gigs)        
If *MemoryID > 0
  FillMemory(*MemoryID,gigs,255,#PB_Byte)  
  If CreateFile(0,GetCurrentDirectory()+"test.tmp")                
    ;FileBuffersSize(0,mysize)                                        
    *writeptr=*MemoryID
    *end = *MemoryID + gigs
    While *writeptr < *end
      If *end-*writeptr >= mysize
        chunksize = mysize
      Else 
        chunksize = *end-*writeptr
      EndIf
      WriteData(0, *writeptr, chunksize)            
      *writeptr + chunksize
    Wend
    ;FlushFileBuffers(0)                     
    CloseFile(0)                             
  EndIf
EndIf
EndProcedure

mytime1=ElapsedMilliseconds()
Netmestro(67108864)
mytime=ElapsedMilliseconds()-mytime1

MessageRequester("mytime",Str(mytime))


; Chunck 512      = 4014 Elapsed time
; Chunck 1024     = 4627 Elapsed time
; Chunck 2048     = 5048 Elapsed time
; Chunck 4096     = 4238 Elapsed time
; Chunck 8192     = 2965 Elapsed time
; Chunck 16384    = 2287 Elapsed time
; Chunck 32768    = 1864 Elapsed time
; Chunck 65536    = 1596 Elapsed time
; Chunck 131072   = 1660 Elapsed time
; Chunck 262144   = 1570 Elapsed time
; Chunck 524288   = 1468 Elapsed time
; Chunck 1048576  = 1499 Elapsed time
; Chunck 2097152  = 1569 Elapsed time
; Chunck 4194304  = 1576 Elapsed time
; Chunck 8388608  = 1468 Elapsed time
; Chunck 16777216 = 1468 Elapsed time
; Chunck 33554432 = 2143 Elapsed time
; Chunck 67108864 = 2529 Elapsed time

if you really want to be precise you can add this...but it is not necessary.

Code: Select all

Repeat
mydimension=FileSize(GetCurrentDirectory()+"test.tmp")   
Until mydimension = 4294967296
User avatar
NicTheQuick
Addict
Addict
Posts: 1223
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by NicTheQuick »

You are also measuring creating and filling the memory with your test. Was that intentional?
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
User avatar
NicTheQuick
Addict
Addict
Posts: 1223
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by NicTheQuick »

However I used your code and tested it against my SSD, NVME and tmpfs (RAM) and I am impressed by your speeds. How do you write 2,7 GB/s to your Harddrive? Are you using a NVME RAID or is it a NVME PCIe 4.0 drive?

Code: Select all

; User             Simo_na       NicTheQuick
; SSD/RAM/NVME               SSD    RAM    NVME
; Chunk 512      = 4014      7692   2721   3222
; Chunk 1024     = 4627      6546   2895   4026
; Chunk 2048     = 5048     10497   2776   3602
; Chunk 4096     = 4238      7744   2546   3210
; Chunk 8192     = 2965     10761   2541   3485
; Chunk 16384    = 2287      8322   2330   3297
; Chunk 32768    = 1864      9978   2255   3288
; Chunk 65536    = 1596      7303   2140   3120
; Chunk 131072   = 1660     10696   2200   3144
; Chunk 262144   = 1570      9361   2068   3057
; Chunk 524288   = 1468      6517   2083   3121
; Chunk 1048576  = 1499      8969   2029   3161
; Chunk 2097152  = 1569      6944   2035   3120
; Chunk 4194304  = 1576      9490   2043   3105
; Chunk 8388608  = 1468      7371   2019   3108
; Chunk 16777216 = 1468      9304   2076   3085
; Chunk 33554432 = 2143      9997   2020   3109
; Chunk 67108864 = 2529      7450   2037   3115
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
User avatar
NicTheQuick
Addict
Addict
Posts: 1223
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by NicTheQuick »

I changed the code so AllocateMemory() gets no longer measured. And now the image is much clearer.

This is the test code:

Code: Select all

EnableExplicit

; The maximum value is: 2 GiB - 1 Byte = (2 * 1024 * 1024 * 1024 - 1) Bytes = (2^31 - 1) Bytes = 2147483647 Bytes
Global MAX_CHUNK_SIZE = 2048 * 1024 * 1024 - 1
Procedure SaveMemory(*Memory, FileName.s, Size = #PB_Ignore)
	Protected result.i, file.i
	Protected chunkSize.i, writtenBytes.i
	
	file = CreateFile(#PB_Any, FileName)
	If Not file
		DebuggerError("Could not open file for writing: " + FileName)
		ProcedureReturn #False
	EndIf
	
	If Size = #PB_Ignore
		Size = MemorySize(*Memory)
	EndIf
	
	While Size
		If Size > MAX_CHUNK_SIZE
			chunkSize = MAX_CHUNK_SIZE
		Else
			chunkSize = Size
		EndIf
			
		writtenBytes = WriteData(file, *Memory, chunkSize)
		If Not writtenBytes
			result = #False
			DebuggerError("Could not write " + chunkSize + " bytes into file.")
			Break
		EndIf
		
		Size - writtenBytes
		*Memory + writtenBytes
	Wend
	
	CloseFile(file)
	
	ProcedureReturn result
EndProcedure

OpenConsole()

Define *mem = AllocateMemory(4 * 1024 * 1024 * 1024)
Define time.i, chunkSizeBits.i, chunkSize.i, file.s, result.s

file.s = "/test/purebasic_memory_test.dat"

PrintN("Chunk Size   Time")

For chunkSizeBits = 9 To 30
	MAX_CHUNK_SIZE = 1 << chunkSizeBits
	
	DeleteFile(file)
	Delay(100)
	
	time = ElapsedMilliseconds()
	SaveMemory(*mem, file)
	time = ElapsedMilliseconds() - time
	
	PrintN(RSet(Str(MAX_CHUNK_SIZE), 10, " ") + "   " + RSet(Str(time), 5, " "))
Next

FreeMemory(*mem)

Input()
CloseConsole()
And now the results for NVME, RAM and SSD, each writing a 4 GiB file.

Code: Select all

; NVME
Chunk Size   Time
       512    2133
      1024    1831
      2048    1626
      4096    1534
      8192    1564
     16384    1453
     32768    1330
     65536    1415
    131072    1294
    262144    1620
    524288    1254
   1048576    1258
   2097152    1512
   4194304    1427
   8388608    1464
  16777216    1453
  33554432    1421
  67108864    1507
 134217728    1401
 268435456    1421
 536870912    1493
1073741824    1352

Code: Select all

; RAM
Chunk Size   Time
       512    1738
      1024    1485
      2048    1244
      4096    1121
      8192    1124
     16384     888
     32768     770
     65536     705
    131072     674
    262144     653
    524288     660
   1048576     648
   2097152     647
   4194304     646
   8388608     643
  16777216     646
  33554432     652
  67108864     646
 134217728     652
 268435456     649
 536870912     644
1073741824     651

Code: Select all

; SSD
Chunk Size   Time
       512    5763
      1024    5353
      2048    6672
      4096    4438
      8192    3536
     16384    4222
     32768    3501
     65536    4736
    131072    7188
    262144    4606
    524288    6663
   1048576    5113
   2097152    5132
   4194304    5569
   8388608    5884
  16777216    7913
  33554432    5448
  67108864    7863
 134217728    5739
 268435456    8021
 536870912    4836
1073741824    5004
As you can see, the NVME gets not much faster after a chunk size of 16 kiB, RAM does not increase anymore after a chunk size of 128 kiB and the SSD is just all over the place.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
Simo_na
Enthusiast
Enthusiast
Posts: 177
Joined: Sun Mar 03, 2013 9:01 am

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Simo_na »

Thank you NicTheQuick, nice !! :)

This is my NVME (WD SN850 - 500GB @ half full...203GB free)

Code: Select all

NVME
 Chunk Size   Time Nic    Chunk Size   Time Simo		
       512    2133	           512    4251
      1024    1831		      1024    4108
      2048    1626		      2048    4829
      4096    1534		      4096    3722
      8192    1564		      8192    2417
     16384    1453		     16384    1500
     32768    1330		     32768    1164
     65536    1415		     65536     922
    131072    1294		    131072     820
    262144    1620		    262144     780
    524288    1254		    524288     775
   1048576    1258		   1048576     750
   2097152    1512		   2097152     716
   4194304    1427		   4194304     700
   8388608    1464		   8388608     697
  16777216    1453		  16777216     686
  33554432    1421		  33554432    1431
  67108864    1507		  67108864    1906
 134217728    1401		 134217728    1993
 268435456    1421		 268435456    2259
 536870912    1493		 536870912    2146
1073741824    1352		1073741824    2573
the 33554432 and 16384 is similar ...
Simo_na
Enthusiast
Enthusiast
Posts: 177
Joined: Sun Mar 03, 2013 9:01 am

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Simo_na »

With imdisk ( :D )
https://sourceforge.net/projects/imdisk-toolkit/

Code: Select all

Chunk Size   Time
       512   12185
      1024   13923
      2048   20464
      4096      70
      8192      48
     16384      37
     32768      29
     65536      24
    131072      20
    262144      18
    524288      16
   1048576      16
   2097152      15
   4194304      15
   8388608      15
  16777216      14
  33554432      18
  67108864      19
 134217728       0
 268435456       0
 536870912       0
1073741824       0
User avatar
NicTheQuick
Addict
Addict
Posts: 1223
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by NicTheQuick »

Simo_na wrote: Thu Jun 30, 2022 6:29 pm With imdisk ( :D )
https://sourceforge.net/projects/imdisk-toolkit/
These values make no sense at all :D
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
Simo_na
Enthusiast
Enthusiast
Posts: 177
Joined: Sun Mar 03, 2013 9:01 am

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Simo_na »

Sure, I was joking..
Thanks for the help.
Post Reply