Page 1 of 1
After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 3:50 pm
by uwekel
Hi,
today i noticed a large memory use in one of my programs. After some tests i found out a strange behavior with the LIST. I wrote a small sample to point out the problem:
Code: Select all
NewList items.s()
MessageRequester("", "Create") ;memory use is now 2.1 MB
For i = 1 To 5000000
AddElement(items())
items() = "This is line number " + Str(i)
Next
MessageRequester("", "Clear") ;memory use is now 384 MB
ClearList(items())
MessageRequester("", "Cleared") ;memory use is still 233 MB
When reaching the third MessageRequester, i expected the same memory consumption as at the beginning, but there are still 233 MB blocked. Why that?
The debugger was disabled. The code ported into Python 2.7.1 works fine (so it's not Fedora or so).
Best regards
Uwe
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 4:12 pm
by falsam
Test : PB 4.61 Window 7
Code: Select all
NewList items.s()
MessageRequester("", "Create") ;memory use is now 1.8 MB
For i = 1 To 5000000
AddElement(items())
items() = "This is line number " + Str(i)
Next
MessageRequester("", "Clear") ;memory use is now 276.6 MB
ClearList(items())
MessageRequester("", "Cleared") ;memory use is still 3.6 MB
FreeList(items()) ;memory use is still 0.3 MB ........YeahhHHHHH :)
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 4:17 pm
by uwekel
I did not mention but i tried already FreeList() with NO effect. The only difference is that the list object itself will be freed. On my system there are still 233 MB used. As you can see in your sample on windows, the main memory consumption must be freed on ClearList().
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 4:34 pm
by uwekel
A test on Fedora 17 32-bit shows the same problem. The maximum memory usage is 262 MB, and will reduce to 195 MB only.
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 4:44 pm
by IdeasVacuum
Interesting one. Your code uwekel works perfectly on Windows XP 32bit, PB4.61, result exactly as expected, Debug off and Debug on.
So, issue is only when on Linux? Any Mac Users out there to test?
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 4:54 pm
by IdeasVacuum
Unexpected result running as a 32bit exe on Win 7 64bit:
Free Ram:
Before 1939MB
List Loaded 1514MB
List Cleared 1949MB
It's OK, but 10MB seems a lot?
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 5:05 pm
by uwekel
The 10 MB is maybe the extra space for the list object. This should be gone after FreeList().
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 5:07 pm
by freak
Never trust programs like the task manager about such things. As long as there is no immediate need for the freed memory, the OS will usually keep it reserved for the program for a possible later allocation. So the number doesn't go down immediately. That doesn't mean that the memory is actually still allocated by the program.
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 5:12 pm
by uwekel
The problem is not caused by the LIST. A test with a string array has a similar result!
Code: Select all
MessageRequester("", "Create") ;memory use is now 2.1 MB
Dim items.s(5000000)
For i = 0 To 5000000-1
items(i) = "This is line number " + Str(i)
Next
MessageRequester("", "Clear") ;memory use is now 269 MB
ReDim items(0)
MessageRequester("", "Cleared") ;memory use is still 231 MB
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 5:15 pm
by uwekel
With Integers it works fine!
But why are the strings not be freed?
Code: Select all
NewList items.i()
MessageRequester("", "Create") ;memory use is now 2.1 MB
For i = 1 To 5000000
AddElement(items())
items() = i
Next
MessageRequester("", "Clear") ;memory use is now 155 MB
ClearList(items())
MessageRequester("", "Cleared") ;memory use is still 4 MB
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 5:17 pm
by IdeasVacuum
Freak is of course right - I don't think the empty List would be 10MB though, that will be something else but not important. I used this to verify the RAM on Windows:
Code: Select all
Import "kernel32.lib"
GlobalMemoryStatusEx.i(*lpBuffer)
EndImport
Define meminfo.MEMORYSTATUSEX
meminfo\dwLength = SizeOf(MEMORYSTATUSEX)
If GlobalMemoryStatusEx(meminfo)
qTotalMem.q = ((meminfo\ullTotalPhys) / 1024 / 1024)
qAvailMem.q = ((meminfo\ullAvailPhys) / 1024 / 1024)
sRamFree.s = "Free RAM: " + Str(qAvailMem) + "MB"
MessageRequester("",sRamFree.s)
EndIf
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 6:27 pm
by uwekel
@freak: I am using the system monitor from Linux to control the memory consumption. In all above cases, the display seems to be correct, except when strings are involved.
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 7:11 pm
by uwekel
To check if the memory consumption increases more and more, i put the codeblock in a loop and the result was very unexpected: the 384 MB would never be exceeded. So the guess of freak is probably right - the memory is reserved for the current program and will be reused! Nice to know
Code: Select all
NewList items.s()
For j= 1 To 3
MessageRequester("", "Create") ;memory use is now 2.1 MB
For i = 1 To 5000000
AddElement(items())
items() = "This is line number " + Str(i)
Next
MessageRequester("", "Clear") ;memory use is now 384 MB
ClearList(items())
MessageRequester("", "Cleared") ;memory use is still 233 MB
Next
Re: After clearing a huge list still a lot of memory in use
Posted: Wed Jul 04, 2012 8:31 pm
by xorc1zt
Code: Select all
; purebasic StringHeap stats
; xorc1zt
; 2012
; windows 7
Structure BlockStr
hMem.i
dwReserved.l[3]
EndStructure
Structure RegionStr
dwCommittedSize.l
dwUnCommittedSize.l
*lpFirstBlock
*lpLastBlock
EndStructure
Structure PROCESS_HEAP_ENTRY
*lpData
cbData.l
cbOverhead.b
iRegionIndex.b
wFlags.w
StructureUnion
Block.BlockStr
Region.RegionStr
EndStructureUnion
EndStructure
Procedure PBStringHeapStats (showRegion.b, showUncommitted.b, showBlock.b)
Define *stringHeap
Define phe.PROCESS_HEAP_ENTRY
Define totalRegion.i = 0
Define totalUncommitted.i = 0
Define totalBlock.i = 0
!extrn PB_StringHeap
!MOV rax, [PB_StringHeap]
!MOV [p.p_stringHeap],rax
Debug "--- StringHeap stats ---"
Debug ""
While (HeapWalk_(*stringHeap, @phe))
If (phe\wFlags & #PROCESS_HEAP_REGION)
totalRegion + 1
If Not (showRegion)
Continue
EndIf
Debug "Region:"
Debug " bytes committed: "+Str(phe\Region\dwCommittedSize)
Debug " bytes uncommitted: "+Str(phe\Region\dwUnCommittedSize)
Debug " First block address: "+Hex(phe\Region\lpFirstBlock)
Debug " Last block address: "+Hex(phe\Region\lpLastBlock)
ElseIf (phe\wFlags & #PROCESS_HEAP_UNCOMMITTED_RANGE)
totalUncommitted + 1
If Not (showUncommitted)
Continue
EndIf
Debug "Uncommitted range:"
Else
totalBlock + 1
If Not (showBlock)
Continue
EndIf
Debug "Block:"
EndIf
Debug " Data portion begins at: "+Hex(phe\lpData)
Debug " Size: "+Str(phe\cbData)+" bytes"
Debug " Overhead: "+Str(phe\cbData)+" bytes"
Debug " Region index: "+Str(phe\iRegionIndex)
Debug ""
Wend
Debug "Total:"
Debug " Regions: "+Str(totalRegion)
Debug " Uncommitteds: "+Str(totalUncommitted)
Debug " Blocks: "+Str(totalBlock)
Debug ""
EndProcedure
; ------------------------------
; ------------ test ------------
; ------------------------------
NewList items.s()
MessageRequester("", "Create") ;memory use is now 1.8 MB
For i = 1 To 5000000
AddElement(items())
items() = "This is line number " + Str(i)
Next
PBStringHeapStats(0,0,0)
MessageRequester("", "Clear") ;memory use is now 276.6 MB
ClearList(items())
PBStringHeapStats(0,0,0)
MessageRequester("", "Cleared") ;memory use is still 3.6 MB
FreeList(items()) ;memory use is still 0.3 MB ........YeahhHHHHH :)