Page 1 of 1

Passing Structure to Threaded Procedures - Memory Leak?

Posted: Mon Dec 22, 2014 6:57 am
by Liqu
Hi, i've been trying to create a procedure that pass a structure to another procedures with structure's pointer but
when i monitored it, the memory usage keep increasing.

Code: Select all

Structure testSTR
  String$
  Semaphore.i
EndStructure

Procedure P1(*TMP.testSTR)
  String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)


EndProcedure

Procedure P2(*TMP.testSTR)
  String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)


EndProcedure




Procedure X(String$)
  
  Protected *sTMP.testSTR
  
  *TMP.testSTR    = AllocateStructure(testSTR)
  *TMP\String$    = String$
  *TMP\Semaphore = CreateSemaphore(1)
  
  
  WaitSemaphore(*TMP\Semaphore)
  CreateThread(@P1(),*TMP)
  WaitSemaphore(*TMP\Semaphore)
  CreateThread(@P2(),*TMP)
  WaitSemaphore(*TMP\Semaphore)
  
  FreeSemaphore(*TMP\Semaphore)
  FreeStructure(*TMP)
EndProcedure


Repeat
  X("TEST")
  Delay(5)
ForEver

Still confused where it leak...

Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Mon Dec 22, 2014 7:37 am
by STARGĂ…TE
I don't see a leak.
Memory usage is constant.

PB 5.30 64x, Unicode

Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Mon Dec 22, 2014 7:57 am
by Liqu
i use 5.31 32bit on x64 machine

after several minutes ( about 5min ) it start growing a little,
then it grow from below 2,000K to above 2,000K
now it's 2,496K

Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Mon Dec 22, 2014 10:47 am
by User_Russian
Check this code.

Code: Select all

Structure testSTR
  String$
  Semaphore.i
EndStructure

Procedure P1(*TMP.testSTR)
  String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)


EndProcedure

Procedure P2(*TMP.testSTR)
  String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)


EndProcedure



Procedure X(String$)
  
  Protected *sTMP.testSTR
  
  *TMP.testSTR    = AllocateMemory(SizeOf(testSTR))
  InitializeStructure(*TMP, testSTR)
  *TMP\String$    = String$
  *TMP\Semaphore = CreateSemaphore(1)
  
  
  WaitSemaphore(*TMP\Semaphore)
  CreateThread(@P1(),*TMP)
  WaitSemaphore(*TMP\Semaphore)
  CreateThread(@P2(),*TMP)
  WaitSemaphore(*TMP\Semaphore)
  
  FreeSemaphore(*TMP\Semaphore)
  ClearStructure(*TMP, testSTR)
  FreeMemory(*TMP)
EndProcedure


Repeat
  X("TEST")
  Delay(5)
ForEver

Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Mon Dec 22, 2014 11:20 am
by Liqu
@User_Russian

i have tested your code
unfortunately it still increased ( Debugger ON with Purifier )

it start from 1,524K and after about 170298 calls ( i checked it with Callstack )
it became 1,904K

Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Mon Dec 22, 2014 11:58 am
by acreis
After a small increase, mem usage is constant.

Create an exe and try.

Best regards

Code: Select all

Prototype.l GetProcessMemoryInfo_(hProcess.l, *p, cb.l)

Procedure.l GetProcessMemoryUsage(kilobytes.b=#True)
  Structure PROCESS_MEMORY_COUNTERS
    cb.l;DW
    PageFaultCount.l;DW
    PeakWorkingSetSize.l;SIZE_T
    WorkingSetSize.l;SIZE_T
    QuotaPeakPagedPoolUsage.l;SIZE_T
    QuotaPagedPoolUsage.l;SIZE_T
    QuotaPeakNonPagedPoolUsage.l;SIZE_T
    QuotaNonPagedPoolUsage.l;SIZE_T
    PagefileUsage.l;SIZE_T
    PeakPagefileUsage.l;SIZE_T
  EndStructure
 
  Static GetProcessMemoryInfo_.GetProcessMemoryInfo_
  Static PSAPI
  If Not GetProcessMemoryInfo_
    If Not PSAPI
      PSAPI=OpenLibrary(#PB_Any, "PSAPI.DLL")
    EndIf
    If PSAPI
      GetProcessMemoryInfo_=GetFunction(PSAPI, "GetProcessMemoryInfo")
    EndIf
  EndIf
  If GetProcessMemoryInfo_
    Protected p.PROCESS_MEMORY_COUNTERS
    GetProcessMemoryInfo_(GetCurrentProcess_(), @p, SizeOf(PROCESS_MEMORY_COUNTERS))
   
    If kilobytes
      ProcedureReturn (p\WorkingSetSize+p\PagefileUsage) /1024
    Else
      ProcedureReturn (p\WorkingSetSize + p\PagefileUsage)
    EndIf
  EndIf
EndProcedure

Structure testSTR
  String$
  Semaphore.i
EndStructure

Procedure P1(*TMP.testSTR)
  String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)


EndProcedure

Procedure P2(*TMP.testSTR)
  String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)


EndProcedure



Procedure X(String$)
  
  Protected *sTMP.testSTR
  
  *TMP.testSTR    = AllocateMemory(SizeOf(testSTR))
  InitializeStructure(*TMP, testSTR)
  *TMP\String$    = String$
  *TMP\Semaphore = CreateSemaphore(1)
  
  
  WaitSemaphore(*TMP\Semaphore)
  CreateThread(@P1(),*TMP)
  WaitSemaphore(*TMP\Semaphore)
  CreateThread(@P2(),*TMP)
  WaitSemaphore(*TMP\Semaphore)
  
  FreeSemaphore(*TMP\Semaphore)
  ClearStructure(*TMP, testSTR)
  FreeMemory(*TMP)
EndProcedure

OpenConsole("mem leak test")
Global count 
Repeat
  X("TEST")
  Delay(1)
  count + 1
  
  If (count % 1000) = 0
    PrintN( Str(count) + "/" + GetProcessMemoryUsage())
  EndIf
  
ForEver


Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Mon Dec 22, 2014 12:51 pm
by luis
Nobody here mentioned if threadsafe has been enabled or not. It should but ...
With threadsafe on, unicode on, debugger on, and a big string ram usage grows quickly.

Win7 x64, pb x86 5.31

Using the code above...

Code: Select all

Prototype.l GetProcessMemoryInfo_(hProcess.l, *p, cb.l)

Procedure.l GetProcessMemoryUsage(kilobytes.b=#True)
  Structure PROCESS_MEMORY_COUNTERS
    cb.l;DW
    PageFaultCount.l;DW
    PeakWorkingSetSize.l;SIZE_T
    WorkingSetSize.l;SIZE_T
    QuotaPeakPagedPoolUsage.l;SIZE_T
    QuotaPagedPoolUsage.l;SIZE_T
    QuotaPeakNonPagedPoolUsage.l;SIZE_T
    QuotaNonPagedPoolUsage.l;SIZE_T
    PagefileUsage.l;SIZE_T
    PeakPagefileUsage.l;SIZE_T
  EndStructure
 
  Static GetProcessMemoryInfo_.GetProcessMemoryInfo_
  Static PSAPI
  If Not GetProcessMemoryInfo_
    If Not PSAPI
      PSAPI=OpenLibrary(#PB_Any, "PSAPI.DLL")
    EndIf
    If PSAPI
      GetProcessMemoryInfo_=GetFunction(PSAPI, "GetProcessMemoryInfo")
    EndIf
  EndIf
  If GetProcessMemoryInfo_
    Protected p.PROCESS_MEMORY_COUNTERS
    GetProcessMemoryInfo_(GetCurrentProcess_(), @p, SizeOf(PROCESS_MEMORY_COUNTERS))
   
    If kilobytes
      ProcedureReturn (p\WorkingSetSize+p\PagefileUsage) /1024
    Else
      ProcedureReturn (p\WorkingSetSize + p\PagefileUsage)
    EndIf
  EndIf
EndProcedure

Structure testSTR
  String$
  Semaphore.i
EndStructure

Procedure P1(*TMP.testSTR)
  String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)
EndProcedure

Procedure P2(*TMP.testSTR)
  String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)
EndProcedure



Procedure X(String$)
  Protected *sTMP.testSTR
 
  *TMP.testSTR    = AllocateMemory(SizeOf(testSTR))
  InitializeStructure(*TMP, testSTR)
  *TMP\String$    = String$
  *TMP\Semaphore = CreateSemaphore(1)
 
 
  WaitSemaphore(*TMP\Semaphore)
  CreateThread(@P1(),*TMP)
  WaitSemaphore(*TMP\Semaphore)
  CreateThread(@P2(),*TMP)
  WaitSemaphore(*TMP\Semaphore)
 
  FreeSemaphore(*TMP\Semaphore)
  ClearStructure(*TMP, testSTR)
  FreeMemory(*TMP)
EndProcedure

s$ = Space(32768); big string

OpenConsole("mem leak test")
Global count
Repeat
  X(s$)
  Delay(1)
  count + 1
 
  If (count % 1000) = 0
    PrintN( Str(count) + "/" + GetProcessMemoryUsage())
  EndIf
 
ForEver
console, debugger on wrote: 3000/25512
4000/25860
5000/26196
6000/26436
7000/26992
8000/26624
9000/26992
10000/26772
11000/27780
12000/28204
13000/28872
14000/28652
15000/29276
16000/29460
17000/29856
18000/30400
19000/31216
20000/31216
21000/31216
22000/31748
23000/32008
24000/32676
25000/33808
26000/34468
27000/34284
28000/34284
29000/34716
30000/35732
31000/35128
32000/36672
33000/36884
34000/37324
35000/37704
36000/38128
37000/38544
38000/38940
39000/39204
40000/39360
41000/39360
42000/39788
43000/39788
44000/40208
45000/40208
46000/41028
47000/41028
48000/41448
...
With debugger off, no increase.
console, debugger off wrote: 1000/5920
2000/6008
3000/6124
4000/6124
5000/6124
6000/6124
7000/6124
8000/6128
9000/6128
10000/6128
11000/6128
12000/6128
13000/6128
14000/6128
15000/6128
16000/6128
17000/6128
18000/6128
19000/6128
20000/6128
21000/6128
22000/6128
23000/6128
24000/6128
25000/6128
26000/6132
27000/6132
28000/6132
29000/6132
30000/6132
31000/6132
32000/6132
33000/6132
34000/6132
35000/6132
36000/6132
37000/6132
38000/6132
39000/6132
40000/6132
41000/6132
42000/6132
43000/6132
44000/6132
45000/6132
46000/6132
47000/6132
48000/6132
49000/6132
50000/6132
51000/6132

If you disable the string copy in the two threaded procs

Code: Select all

Procedure P1(*TMP.testSTR)
  ;String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)
EndProcedure

Procedure P2(*TMP.testSTR)
  ;String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)
EndProcedure
then even with the debugger enabled ram usage doesn't grow.

So the cause seems to be located there but only when debugger is active.

Another strange thing... even with the two string copy above commented out, using AllocateStructure() instead of AllocateMemory()/InitializeStructure() gives a small leakage with the debugger.

So in short:

debugger off, no problem
debugger on, problem with the two string copy in the threaded procs and also some kind of leakage with AllocateStructure()

Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Mon Dec 22, 2014 3:44 pm
by ostapas
You can use RemovePagefaults() procedure from PBOSL lib, not an elegant solution, but usually helps when you have constantly growing memory usage, but have no time or desire to debug it :)

Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Tue Dec 23, 2014 12:11 am
by luis
ostapas wrote:You can use RemovePagefaults() procedure from PBOSL lib, not an elegant solution, but usually helps when you have constantly growing memory usage, but have no time or desire to debug it :)
Since RemovePagefaults() (???) uses EmptyWorkingSet(), afaik this just "visually" hides the problem, if any, resulting in a smaller number in the column "private working set" inside the task manager.

If the memory used by a process grows too much (for a leakage or for legitimate reasons) the OS will remove the less recent used pages from the working set of the process by itself. The pages are moved to a standby list, and are actually cached there (so still in ram) to be given back to the original process if required.
But if the free memory is low and another process is requiring ram to run, the page is actually removed from the cache and the freed ram assigned to the other process working set which actually needs it right now.

EmptyWorkingSet() just does that sooner than later, but does not cure a leakage. The allocated page produced by the leakage is just moved sooner to the standby lists, and consequently the size of the private working set is reported as smaller in the task manager. The ram is still allocated, it's just not listed under the private working set anymore, nothing really changes.
Only personal satisfaction in seeing a small number.

The only difference is EmptyWorkingSet() will cause a soft page fault if the page is needed again by the original process in order to retrieve it from the standby list, instead of simply having the page already there if enough ram was available.
Worst case can cause an hard page fault if the physical memory associated to the page prematurely moved to the standby list has been assigned to another process, so the original page contents have been swapped out and now the uncached page is requested again by the original process.
In short, I think it can (even if it's unlikely) be detrimental for a program not leaking (swapped pages can legitimately be requested again), and probably irrelevant for a leaking program (those page would be swapped out anyway if and when really needed).

One legitimate application-level use I can think for EmptyWorkingSet() is for a program requiring a lot of resources but for short interval of time.
For example it waits for something to happen, then process a huge amount of data for that event, but the next event is not expected for some time.
It may invoke EmptyWorkingSet() to make available its pages not needed anymore to other programs without the need to wait for the OS to do so.
Maybe this could be an a good idea. Maybe.

Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Tue Dec 23, 2014 6:50 am
by Liqu
Debugger OFF, Threadsafe ON, Unicode ON, PB 32bit, W7 x64

AllocateStructure, FreeStructure
6225000/7060

AllocateMemory, InitializeStructure, ClearStructure, FreeMemory
6118000/7068

Code: Select all

Prototype.l GetProcessMemoryInfo_(hProcess.l, *p, cb.l)

Procedure.l GetProcessMemoryUsage(kilobytes.b=#True)
  Structure PROCESS_MEMORY_COUNTERS
    cb.l;DW
    PageFaultCount.l;DW
    PeakWorkingSetSize.l;SIZE_T
    WorkingSetSize.l;SIZE_T
    QuotaPeakPagedPoolUsage.l;SIZE_T
    QuotaPagedPoolUsage.l;SIZE_T
    QuotaPeakNonPagedPoolUsage.l;SIZE_T
    QuotaNonPagedPoolUsage.l;SIZE_T
    PagefileUsage.l;SIZE_T
    PeakPagefileUsage.l;SIZE_T
  EndStructure
 
  Static GetProcessMemoryInfo_.GetProcessMemoryInfo_
  Static PSAPI
  If Not GetProcessMemoryInfo_
    If Not PSAPI
      PSAPI=OpenLibrary(#PB_Any, "PSAPI.DLL")
    EndIf
    If PSAPI
      GetProcessMemoryInfo_=GetFunction(PSAPI, "GetProcessMemoryInfo")
    EndIf
  EndIf
  If GetProcessMemoryInfo_
    Protected p.PROCESS_MEMORY_COUNTERS
    GetProcessMemoryInfo_(GetCurrentProcess_(), @p, SizeOf(PROCESS_MEMORY_COUNTERS))
   
    If kilobytes
      ProcedureReturn (p\WorkingSetSize+p\PagefileUsage) /1024
    Else
      ProcedureReturn (p\WorkingSetSize + p\PagefileUsage)
    EndIf
  EndIf
EndProcedure

Structure testSTR
  String$
  Semaphore.i
EndStructure

Procedure P1(*TMP.testSTR)
  String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)
EndProcedure

Procedure P2(*TMP.testSTR)
  String$ = *TMP\String$
  SignalSemaphore(*TMP\Semaphore)
EndProcedure



Procedure X(String$)
  Protected *sTMP.testSTR
 
  *TMP.testSTR    = AllocateMemory(SizeOf(testSTR))
  InitializeStructure(*TMP, testSTR)
  *TMP\String$    = String$
  *TMP\Semaphore = CreateSemaphore(1)
 
 
  WaitSemaphore(*TMP\Semaphore)
  CreateThread(@P1(),*TMP)
  WaitSemaphore(*TMP\Semaphore)
  CreateThread(@P2(),*TMP)
  WaitSemaphore(*TMP\Semaphore)
 
  FreeSemaphore(*TMP\Semaphore)
  ClearStructure(*TMP, testSTR)
  FreeMemory(*TMP)
EndProcedure

s$ = Space(32768); big string

OpenConsole("mem leak test")
Global count
Repeat
  X(s$)
  Delay(1)
  count + 1
 
  If (count % 1000) = 0
    PrintN( Str(count) + "/" + GetProcessMemoryUsage())
  EndIf
 
ForEver

Thank you very much for the help :)

Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Tue Dec 23, 2014 11:45 am
by ostapas
Also, many thanks to Luis for such elaborate explanation!

Re: Passing Structure to Threaded Procedures - Memory Leak?

Posted: Wed Dec 24, 2014 6:32 pm
by Liqu
@ostapas
yeah, i've learned a lot too :)