After clearing a huge list still a lot of memory in use

[uwekel]

Hi,

today i noticed a large memory use in one of my programs. After some tests i found out a strange behavior with the LIST. I wrote a small sample to point out the problem:

Code: Select all

NewList items.s()

MessageRequester("", "Create") ;memory use is now 2.1 MB

For i = 1 To 5000000
  AddElement(items())
  items() = "This is line number " + Str(i)
Next

MessageRequester("", "Clear") ;memory use is now 384 MB

ClearList(items())

MessageRequester("", "Cleared")  ;memory use is still 233 MB

When reaching the third MessageRequester, i expected the same memory consumption as at the beginning, but there are still 233 MB blocked. Why that?

The debugger was disabled. The code ported into Python 2.7.1 works fine (so it's not Fedora or so).

Best regards
Uwe

[falsam]

Test : PB 4.61 Window 7

Code: Select all

NewList items.s()

MessageRequester("", "Create") ;memory use is now 1.8 MB

For i = 1 To 5000000
  AddElement(items())
  items() = "This is line number " + Str(i)
Next

MessageRequester("", "Clear") ;memory use is now 276.6 MB

ClearList(items())

MessageRequester("", "Cleared")  ;memory use is still 3.6 MB

FreeList(items()) ;memory use is still 0.3 MB ........YeahhHHHHH :)

http://www.purebasic.com/documentation/ ... elist.html

[uwekel]

I did not mention but i tried already FreeList() with NO effect. The only difference is that the list object itself will be freed. On my system there are still 233 MB used. As you can see in your sample on windows, the main memory consumption must be freed on ClearList().

[uwekel]

A test on Fedora 17 32-bit shows the same problem. The maximum memory usage is 262 MB, and will reduce to 195 MB only.

[IdeasVacuum]

Interesting one. Your code uwekel works perfectly on Windows XP 32bit, PB4.61, result exactly as expected, Debug off and Debug on.

So, issue is only when on Linux? Any Mac Users out there to test?

[IdeasVacuum]

Unexpected result running as a 32bit exe on Win 7 64bit:
Free Ram:
Before 1939MB
List Loaded 1514MB
List Cleared 1949MB

It's OK, but 10MB seems a lot?

[uwekel]

The 10 MB is maybe the extra space for the list object. This should be gone after FreeList().

[freak]

Never trust programs like the task manager about such things. As long as there is no immediate need for the freed memory, the OS will usually keep it reserved for the program for a possible later allocation. So the number doesn't go down immediately. That doesn't mean that the memory is actually still allocated by the program.

[uwekel]

The problem is not caused by the LIST. A test with a string array has a similar result!

Code: Select all

MessageRequester("", "Create") ;memory use is now 2.1 MB

Dim items.s(5000000)

For i = 0 To 5000000-1
  items(i) = "This is line number " + Str(i)
Next

MessageRequester("", "Clear") ;memory use is now 269 MB

ReDim items(0)

MessageRequester("", "Cleared")  ;memory use is still 231 MB

[uwekel]

With Integers it works fine!
But why are the strings not be freed?

Code: Select all

NewList items.i()

MessageRequester("", "Create") ;memory use is now 2.1 MB

For i = 1 To 5000000
  AddElement(items())
  items() = i
Next

MessageRequester("", "Clear") ;memory use is now 155 MB

ClearList(items())

MessageRequester("", "Cleared")  ;memory use is still 4 MB

[IdeasVacuum]

Freak is of course right - I don't think the empty List would be 10MB though, that will be something else but not important. I used this to verify the RAM on Windows:

Code: Select all

       Import "kernel32.lib"
                  GlobalMemoryStatusEx.i(*lpBuffer)
       EndImport

       Define meminfo.MEMORYSTATUSEX
                  meminfo\dwLength = SizeOf(MEMORYSTATUSEX)

       If GlobalMemoryStatusEx(meminfo)

                     qTotalMem.q = ((meminfo\ullTotalPhys) / 1024 / 1024)
                    qAvailMem.q = ((meminfo\ullAvailPhys) / 1024 / 1024)
                     sRamFree.s = "Free RAM: " + Str(qAvailMem) + "MB"
                     MessageRequester("",sRamFree.s)
       EndIf

[uwekel]

@freak: I am using the system monitor from Linux to control the memory consumption. In all above cases, the display seems to be correct, except when strings are involved.

[uwekel]

To check if the memory consumption increases more and more, i put the codeblock in a loop and the result was very unexpected: the 384 MB would never be exceeded. So the guess of freak is probably right - the memory is reserved for the current program and will be reused! Nice to know

Code: Select all

NewList items.s()

For j= 1 To 3
  
  MessageRequester("", "Create") ;memory use is now 2.1 MB
  
  For i = 1 To 5000000
    AddElement(items())
    items() = "This is line number " + Str(i)
  Next
  
  MessageRequester("", "Clear") ;memory use is now 384 MB
  
  ClearList(items())
  
  MessageRequester("", "Cleared")  ;memory use is still 233 MB

Next

[xorc1zt]

Code: Select all

; purebasic StringHeap stats
; xorc1zt
; 2012
; windows 7

Structure BlockStr
    hMem.i
    dwReserved.l[3]
EndStructure

Structure RegionStr
    dwCommittedSize.l
    dwUnCommittedSize.l
    *lpFirstBlock
    *lpLastBlock
EndStructure

Structure PROCESS_HEAP_ENTRY
    *lpData
    cbData.l
    cbOverhead.b
    iRegionIndex.b
    wFlags.w
    StructureUnion
        Block.BlockStr
        Region.RegionStr
    EndStructureUnion
EndStructure

Procedure PBStringHeapStats (showRegion.b, showUncommitted.b, showBlock.b)
    Define *stringHeap
    Define phe.PROCESS_HEAP_ENTRY
    
    Define totalRegion.i      = 0
    Define totalUncommitted.i = 0
    Define totalBlock.i       = 0
    
    !extrn PB_StringHeap
    !MOV rax, [PB_StringHeap]
    !MOV [p.p_stringHeap],rax 
    
    Debug "--- StringHeap stats ---"
    Debug ""
    
    While (HeapWalk_(*stringHeap, @phe))
        If (phe\wFlags & #PROCESS_HEAP_REGION)   
            totalRegion + 1
            
            If Not (showRegion)
                Continue
            EndIf
            
            Debug "Region:"
            Debug "    bytes committed: "+Str(phe\Region\dwCommittedSize)
            Debug "    bytes uncommitted: "+Str(phe\Region\dwUnCommittedSize)
            Debug "    First block address: "+Hex(phe\Region\lpFirstBlock)
            Debug "    Last block address: "+Hex(phe\Region\lpLastBlock)
        ElseIf (phe\wFlags & #PROCESS_HEAP_UNCOMMITTED_RANGE)      
            totalUncommitted + 1
            
            If Not (showUncommitted)
                Continue
            EndIf
            
            Debug "Uncommitted range:"
        Else
            totalBlock + 1
            
            If Not (showBlock)
                Continue
            EndIf
            
            Debug "Block:"
        EndIf
        
        Debug "    Data portion begins at: "+Hex(phe\lpData)
        Debug "    Size: "+Str(phe\cbData)+" bytes"
        Debug "    Overhead: "+Str(phe\cbData)+" bytes"
        Debug "    Region index: "+Str(phe\iRegionIndex)
        Debug ""
    Wend
    
    Debug "Total:"
    Debug "    Regions: "+Str(totalRegion)
    Debug "    Uncommitteds: "+Str(totalUncommitted)
    Debug "    Blocks: "+Str(totalBlock)
    Debug ""
EndProcedure


; ------------------------------
; ------------ test ------------
; ------------------------------

NewList items.s()


MessageRequester("", "Create") ;memory use is now 1.8 MB

For i = 1 To 5000000
  AddElement(items())
  items() = "This is line number " + Str(i)
Next

PBStringHeapStats(0,0,0)
MessageRequester("", "Clear") ;memory use is now 276.6 MB
ClearList(items())
PBStringHeapStats(0,0,0)
MessageRequester("", "Cleared")  ;memory use is still 3.6 MB
FreeList(items()) ;memory use is still 0.3 MB ........YeahhHHHHH :)