It is currently Fri Apr 10, 2020 11:08 am

All times are UTC + 1 hour




Post new topic Reply to topic  [ 37 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: [Done] Free Strings / Memory
PostPosted: Tue Feb 06, 2018 5:07 pm 
Offline
User
User

Joined: Sat Jan 20, 2018 5:28 pm
Posts: 29
Location: Germany
PureBasic does not release the memory completely(only half size), when a string variable is set to #Null$.
In the example below, more than 510MB Ram is still consumed after the string is released.

from the help file:
Quote:
#Null$ : represents an null string. This can be used for API functions requiring a null pointer to a string, or to really free a string.
But ~510mb is not really free!

My Setup
  • PureBasic 5.61 (Windows - x64)
  • Windows 7 x64

This is only a simply/silly example, to simulate the problem. (on my system: MemoryUsage: 519.956 K)
Code:
EnableExplicit
Prototype.i GetProcessMemoryInfo(hProcess.i, *ppsmemCounters.PROCESS_MEMORY_COUNTERS, cb.i)
Procedure.i GetProcessMemoryUsage(hPID)
   Protected Result.i, PMC.PROCESS_MEMORY_COUNTERS, GetProcessMemoryInfo.GetProcessMemoryInfo, Lib = OpenLibrary(#PB_Any, "psapi.dll")
   If Lib
      GetProcessMemoryInfo.GetProcessMemoryInfo = GetFunction(Lib, "GetProcessMemoryInfo")
      If GetProcessMemoryInfo(hPID, @PMC, SizeOf(PROCESS_MEMORY_COUNTERS))
         Result = PMC\WorkingSetSize
      EndIf
      CloseLibrary(Lib)
   EndIf
   ProcedureReturn Result / 1024
EndProcedure

Procedure.s Msg(Text.s="", ToMsgBox.b=1)
   Static PID = 0
   If PID = 0 : PID = GetCurrentProcess_() : EndIf

   Protected MemSize.s = "MemoryUsage: " + FormatNumber(GetProcessMemoryUsage(PID), 0, ",", ".") + " K"
   If ToMsgBox
      If MessageRequester("Info", Text + ~"\n\n" + MemSize, 1) <> 1 : End : EndIf
   Else
      ProcedureReturn MemSize
   EndIf
EndProcedure

Msg("create MyStr now")
Define MyStr.s = Space(1024*1024*250) ; Simulation, we work hard...
Msg("MyStr created, now set MyStr to #Null$")
MyStr = #Null$ ; Done, no need this string onymore

If OpenWindow(0, 0, 0, 300, 35, "memory usage", #PB_Window_MinimizeGadget|#PB_Window_MaximizeGadget|#PB_Window_ScreenCentered|#PB_Window_SizeGadget)
   TextGadget(0, 10, 10, 290, 20, "")
   AddWindowTimer(0, 111, 250)
   Repeat
      Select WaitWindowEvent()
         Case #PB_Event_Timer
            If EventTimer() = 111
               SetGadgetText(0, Msg("", 0))
            EndIf
         Case #PB_Event_CloseWindow
            CloseWindow(0)
            Break
      EndSelect
   ForEver
EndIf
Msg("EndOfFile")


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Tue Feb 06, 2018 6:33 pm 
Offline
Addict
Addict
User avatar

Joined: Sat Feb 13, 2010 3:45 pm
Posts: 1040
As you can see in the following example, memory doesn't grow, if you repeat your simulation.

Code:
EnableExplicit
Prototype.i GetProcessMemoryInfo(hProcess.i, *ppsmemCounters.PROCESS_MEMORY_COUNTERS, cb.i)
Procedure.i GetProcessMemoryUsage(hPID)
   Protected Result.i, PMC.PROCESS_MEMORY_COUNTERS, GetProcessMemoryInfo.GetProcessMemoryInfo, Lib = OpenLibrary(#PB_Any, "psapi.dll")
   If Lib
      GetProcessMemoryInfo.GetProcessMemoryInfo = GetFunction(Lib, "GetProcessMemoryInfo")
      If GetProcessMemoryInfo(hPID, @PMC, SizeOf(PROCESS_MEMORY_COUNTERS))
         Result = PMC\WorkingSetSize
      EndIf
      CloseLibrary(Lib)
   EndIf
   ProcedureReturn Result / 1024
EndProcedure

Procedure.s Msg(Text.s="", ToMsgBox.b=1)
   Static PID = 0
   If PID = 0 : PID = GetCurrentProcess_() : EndIf

   Protected MemSize.s = "MemoryUsage: " + FormatNumber(GetProcessMemoryUsage(PID), 0, ",", ".") + " K"
   If ToMsgBox
      If MessageRequester("Info", Text + ~"\n\n" + MemSize, 1) <> 1 : End : EndIf
   Else
      ProcedureReturn MemSize
   EndIf
EndProcedure

Define MyStr.s
Msg("Starting")

MyStr = Space(1024*1024*250) ; Simulation, we work hard...
Msg("MyStr created")

MyStr = #Null$ ; Done, no need this string onymore
Msg("MyStr set to #Null$")

MyStr = Space(1024*1024*250) ; Simulation, we work hard...
Msg("MyStr created")

MyStr = #Null$ ; Done, no need this string onymore
Msg("MyStr set to #Null$")

MyStr = Space(1024*1024*250) ; Simulation, we work hard...
Msg("MyStr created")

MyStr = #Null$ ; Done, no need this string onymore
Msg("MyStr set to #Null$")


I don't know how that works, but I can well imagine that PB doesn't return all of the requested memory to Windows, as there is a chance that once again a larger amount of memory will be requested.

_________________
sorry for my bad english


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Tue Feb 06, 2018 9:11 pm 
Offline
User
User

Joined: Sat Jan 20, 2018 5:28 pm
Posts: 29
Location: Germany
Yes that's right, but not the point. I want to release the memory because I just do not need it anymore.
That should work: MyStr = #Null$ (- Added: #Null$ special string constant to affect a real null value to a string (to free it completely).)

For example: I wrote a program that only performs a very large operation at startup.(several hundreds of mb.)
After that, only small processes are done by a few bytes. Now my program consumes many GB of memory, though 20mb are enough?

There must be a way to free the unused ram again.
That should also be possible, with #Null$, says the help. And that is exactly what does not work here in this example.

Something's wrong with the Dynamic String Memory Reservation.
Here's an example where the ram consumption grows up to 8GB, which is unlikely to happen.

Code:
EnableExplicit
Prototype.i GetProcessMemoryInfo(hProcess.i, *ppsmemCounters.PROCESS_MEMORY_COUNTERS, cb.i)
Procedure.i GetProcessMemoryUsage(hPID)
   Protected Result.i, PMC.PROCESS_MEMORY_COUNTERS, GetProcessMemoryInfo.GetProcessMemoryInfo
   Protected Lib = OpenLibrary(#PB_Any, "psapi.dll")
   If Lib
      GetProcessMemoryInfo.GetProcessMemoryInfo = GetFunction(Lib, "GetProcessMemoryInfo")
      If GetProcessMemoryInfo(hPID, @PMC, SizeOf(PROCESS_MEMORY_COUNTERS))
         Result = PMC\WorkingSetSize
      EndIf
      CloseLibrary(Lib)
   EndIf
   ProcedureReturn Result / 1024
EndProcedure

Procedure Msg(Loop, Time)
   Static PID = 0
   If PID = 0 : PID = GetCurrentProcess_() : EndIf
   PrintN(RSet(Str(Loop), 7) + #TAB$ + RSet(FormatNumber(GetProcessMemoryUsage(PID), 0, ",", ".") + " K", 16) + #TAB$ + RSet(Str(Time), 10) + "ms")
EndProcedure

If OpenConsole()
   Define i, elemente = 1000000, fill$ = Space(1000000), Dim Zeile$(elemente), Timer.q = ElapsedMilliseconds(), TimeT.q = Timer
   PrintN("   Loops      MemoryUsage      duration for 10000 passes")
   For i = 0 To elemente
      If i % 10000 = 0
         Msg(i, ElapsedMilliseconds()-Timer)
         Timer = ElapsedMilliseconds()
      EndIf
      Zeile$(i) = fill$
      ;Zeile$(i) = #Null$ ; uncommend this in the "second" run, to see the difference. (#Null$ works) and ram usage is OK.
      Zeile$(i) = "a"
   Next i
   PrintN("Total Time: " + Str((ElapsedMilliseconds()-TimeT)/ 1000) + " seconds")
   Input()
EndIf
Result:
Code:
   Loops      MemoryUsage      duration for 10000 passes
      0         13.508 K                 0ms
  10000         92.400 K             30310ms
...
 200000      1.590.872 K             29879ms
...
 350000      2.773.852 K             31691ms
...
 500000      3.956.820 K             31319ms
...
1000000      7.900.064 K             32853ms
Total Time: 3166 seconds

And then run the example one more time, but uncommend this: ";Zeile$(i) = #Null$"
Here also works #Null$, and the MemoryUsage is ok
Edit: But #Null$ should NOT really be required here, so a bug with Dynamic String Memory Reservation!

PS: Purebasic is in this example very, very slow... 53 minutes, I wrote the same script in autoit, and was ready after 12 minutes :shock:


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Tue Feb 06, 2018 10:30 pm 
Offline
Administrator
Administrator

Joined: Fri May 17, 2002 4:39 pm
Posts: 14021
Location: France
What you are experimenting is memory fragmentation: when you ask for a block of 100000 bytes, it needs to be 1000000 bytes of continuous memory. If you just affected "a" in it, the max block you can allocate is now 999999 bytes and Windows will let choose another 1000000 block. There is no bug in string memory allocation, it's just HeapAlloc() under the hood.


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Tue Feb 06, 2018 10:48 pm 
Offline
User
User

Joined: Sat Jan 20, 2018 5:28 pm
Posts: 29
Location: Germany
Fred wrote:
What you are experimenting is memory fragmentation: when you ask for a block of 100000 bytes, it needs to be 1000000 bytes of continuous memory. If you just affected "a" in it, the max block you can allocate is now 999999 bytes and Windows will let choose another 1000000 block. There is no bug in string memory allocation, it's just HeapAlloc() under the hood.

And what you say to my first post?


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Wed Feb 07, 2018 10:13 am 
Offline
Administrator
Administrator

Joined: Fri May 17, 2002 4:39 pm
Posts: 14021
Location: France
PB uses an internal working buffer which is equal to the largest string to allow fast string concat and chaining operation. This temp buffer is readjusted dynamically after a few call if the next string operations doesn't involve big strings anymore.


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Wed Feb 07, 2018 10:19 am 
Offline
Enthusiast
Enthusiast
User avatar

Joined: Sun Jun 11, 2006 12:07 am
Posts: 554
Location: Near Hamburg | Status: Currently not active in programming.
Thanks for the information, Fred.

Is there a way to force PB to release all the internal String-memory?

_________________
PB 5.71 x64, OS: Windows 7 Pro x64, Desktopscaling: 125%, CPU: I7 6500, RAM: 16 GB, GPU: Intel Graphics HD 520, User age: 51y
"Happiness is a pet." | "Never run a changing system!"


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Wed Feb 07, 2018 10:59 am 
Offline
Administrator
Administrator

Joined: Fri May 17, 2002 4:39 pm
Posts: 14021
Location: France
No, it's not available. You should avoid very big string anyway, PB struggle to handle them correctly. Better work with a memory buffer and a string builder like your can find here for example: viewtopic.php?f=12&t=69079


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Wed Feb 07, 2018 11:02 am 
Offline
Enthusiast
Enthusiast
User avatar

Joined: Sun Jun 11, 2006 12:07 am
Posts: 554
Location: Near Hamburg | Status: Currently not active in programming.
Thanks for your assessment. In my case, the question was only asked out of curiosity.

_________________
PB 5.71 x64, OS: Windows 7 Pro x64, Desktopscaling: 125%, CPU: I7 6500, RAM: 16 GB, GPU: Intel Graphics HD 520, User age: 51y
"Happiness is a pet." | "Never run a changing system!"


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Wed Feb 07, 2018 11:59 am 
Offline
User
User

Joined: Sat Jan 20, 2018 5:28 pm
Posts: 29
Location: Germany
Thanks for the info Fred.
Fred wrote:
This temp buffer is readjusted dynamically after a few call if the next string operations doesn't involve big strings anymore.

Could it be provoked to trigger this?


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Thu Feb 08, 2018 7:47 pm 
Offline
Addict
Addict
User avatar

Joined: Wed Jun 11, 2003 9:33 pm
Posts: 4612
Location: Spa, relaxing and thinking, and learning...
There is a trick by Timo 'Freak' time ago to free strings (I am not able to find out in the forum).
The trick is just this one:
Code:
Procedure FreePBString(*Address)
  Protected String.String  ; the String Structure contains one String element, which is initialized to 0 on procedure start
  PokeL(@String,*Address) ; poke our strings address into the structure
EndProcedure               ; when returning, PB's string free routine for the local structure will actually free the passed string.

_________________
http://www.zeitgeistmovie.com

While world=business:world+mafia:Wend


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Thu Feb 08, 2018 8:33 pm 
Offline
User
User

Joined: Sat Jan 20, 2018 5:28 pm
Posts: 29
Location: Germany
Thanks, but it does not work. Am I doing this wrong?

Code:
EnableExplicit
Prototype.i GetProcessMemoryInfo(hProcess.i, *ppsmemCounters.PROCESS_MEMORY_COUNTERS, cb.i)
Procedure.i GetProcessMemoryUsage(hPID)
   Protected Result.i, PMC.PROCESS_MEMORY_COUNTERS, GetProcessMemoryInfo.GetProcessMemoryInfo, Lib = OpenLibrary(#PB_Any, "psapi.dll")
   If Lib
      GetProcessMemoryInfo.GetProcessMemoryInfo = GetFunction(Lib, "GetProcessMemoryInfo")
      If GetProcessMemoryInfo(hPID, @PMC, SizeOf(PROCESS_MEMORY_COUNTERS))
         Result = PMC\WorkingSetSize
      EndIf
      CloseLibrary(Lib)
   EndIf
   ProcedureReturn Result / 1024
EndProcedure
Procedure.s Msg(Text.s="", ToMsgBox.b=1)
   Static PID = 0
   If PID = 0 : PID = GetCurrentProcess_() : EndIf
   Protected MemSize.s = "MemoryUsage: " + FormatNumber(GetProcessMemoryUsage(PID), 0, ",", ".") + " K"
   If ToMsgBox
      If MessageRequester("Info", Text + ~"\n\n" + MemSize, 1) <> 1 : End : EndIf
   Else
      ProcedureReturn MemSize
   EndIf
EndProcedure
Procedure FreePBString(*Address)
  Protected String.String  ; the String Structure contains one String element, which is initialized to 0 on procedure start
  PokeL(@String, *Address) ; poke our strings address into the structure
EndProcedure               ; when returning, PB's string free routine for the local structure will actually free the passed string.

; It does not work that way.
Msg("create MyStr now")
Define i, MyStr.s = Space(1024*1024*250) ; Simulation, we work hard...
Msg("MyStr created, now call FreePBString")
FreePBString(@MyStr)

; It does not work that way either.
; Msg("create MyStr now")
; Define *MyStr.String = AllocateStructure(String)
; *MyStr\s = Space(1024*1024*250) ; Simulation, we work hard...
; Msg("MyStr created, now call FreePBString")
; FreePBString(@*MyStr\s)

If OpenWindow(0, 0, 0, 300, 35, "memory usage", #PB_Window_MinimizeGadget|#PB_Window_MaximizeGadget|#PB_Window_ScreenCentered|#PB_Window_SizeGadget)
   TextGadget(0, 10, 10, 290, 20, "") : AddWindowTimer(0, 111, 250)
   Repeat
      Select WaitWindowEvent()
         Case #PB_Event_Timer
            If EventTimer() = 111
               SetGadgetText(0, Msg("", 0))
            EndIf
         Case #PB_Event_CloseWindow
            CloseWindow(0)
            Break
      EndSelect
   ForEver
EndIf
Msg("EndOfFile")


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Fri Feb 09, 2018 5:31 pm 
Offline
Addict
Addict
User avatar

Joined: Tue Nov 09, 2010 10:15 pm
Posts: 1626
If you only need it temporarily at startup, why not use a local variable in a procedure? then when it is done, it should be free.


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Fri Feb 09, 2018 6:28 pm 
Offline
Enthusiast
Enthusiast

Joined: Wed Nov 09, 2011 8:58 am
Posts: 579
Psychophanta wrote:
There is a trick by Timo 'Freak' time ago to free strings (I am not able to find out in the forum).
The trick is just this one:
Code:
Procedure FreePBString(*Address)
  Protected String.String  ; the String Structure contains one String element, which is initialized to 0 on procedure start
  PokeL(@String,*Address) ; poke our strings address into the structure
EndProcedure               ; when returning, PB's string free routine for the local structure will actually free the passed string.




This post? viewtopic.php?p=156449#p156449

_________________
My new project: A simple HTML/CSS UI engine for PB desktop apps
viewtopic.php?f=7&t=74582


Top
 Profile  
Reply with quote  
 Post subject: Re: Free Strings / Memory
PostPosted: Fri Feb 09, 2018 6:57 pm 
Offline
User
User

Joined: Sat Jan 20, 2018 5:28 pm
Posts: 29
Location: Germany
@Tenaja
That's exactly what PB does not do. Show example, ram usage ~520MB
Code:
EnableExplicit
Prototype.i GetProcessMemoryInfo(hProcess.i, *ppsmemCounters.PROCESS_MEMORY_COUNTERS, cb.i)
Procedure.i GetProcessMemoryUsage(hPID)
   Protected Result.i, PMC.PROCESS_MEMORY_COUNTERS, GetProcessMemoryInfo.GetProcessMemoryInfo, Lib = OpenLibrary(#PB_Any, "psapi.dll")
   If Lib
      GetProcessMemoryInfo.GetProcessMemoryInfo = GetFunction(Lib, "GetProcessMemoryInfo")
      If GetProcessMemoryInfo(hPID, @PMC, SizeOf(PROCESS_MEMORY_COUNTERS))
         Result = PMC\WorkingSetSize
      EndIf
      CloseLibrary(Lib)
   EndIf
   ProcedureReturn Result / 1024
EndProcedure
Procedure.s Msg(Text.s="", ToMsgBox.b=1)
   Static PID = 0
   If PID = 0 : PID = GetCurrentProcess_() : EndIf
   Protected MemSize.s = "MemoryUsage: " + FormatNumber(GetProcessMemoryUsage(PID), 0, ",", ".") + " K"
   If ToMsgBox
      If MessageRequester("Info", Text + ~"\n\n" + MemSize, 1) <> 1 : End : EndIf
   Else
      ProcedureReturn MemSize
   EndIf
EndProcedure

Procedure MyTest()
   Protected MyStr.s
   Msg("create MyStr in procedure now")
   MyStr = Space(1024*1024*250) ; Simulation, we work hard...
   Msg("MyStr in procedure created, now set MyStr = #Null$")
   MyStr = #Null$
EndProcedure

; Call MyTest
MyTest()

If OpenWindow(0, 0, 0, 300, 35, "memory usage", #PB_Window_MinimizeGadget|#PB_Window_MaximizeGadget|#PB_Window_ScreenCentered|#PB_Window_SizeGadget)
   TextGadget(0, 10, 10, 290, 20, "") : AddWindowTimer(0, 111, 250)
   Repeat
      Select WaitWindowEvent()
         Case #PB_Event_Timer
            If EventTimer() = 111
               SetGadgetText(0, Msg("", 0))
            EndIf
         Case #PB_Event_CloseWindow
            CloseWindow(0)
            Break
      EndSelect
   ForEver
EndIf
Msg("EndOfFile")


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 37 posts ]  Go to page 1, 2, 3  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

 


Powered by phpBB © 2008 phpBB Group
subSilver+ theme by Canver Software, sponsor Sanal Modifiye