Memory usage and thread

[tatanas]

Hi,

I'm coding a tcp server on Debian and I'm hunting memory "leak". I create one thread per connected client and I could have 300 clients simultaneous.

I would like to know if I can trust the RES column (htop for example or task manager) indicating the memory usage of my program.
In fact, when the client thread cleanly closes, the memory usage doesn't decrease.

Here is a sample with threads and multiple declared structures inside :

Code: Select all

Structure Person ;(208 bytes)
	a.i
	b.i
	c.i
	d.i
	e.i
	f.i
	g.i
	h.i
	i.i
	j.i
	k.i
	l.i
	m.i
	n.i
	o.i
	p.i
	q.i
	r.i
	s.i
	t.i
	u.i
	v.i
	w.i
	x.i
	y.i
	z.i
EndStructure


ImportC ""
	fopen(filename.p-utf8, mode.p-ascii)
	fscanf(stream.i, format.i, size.i)
	fclose(fp.i)
	getpagesize()
EndImport

Global SC_PAGESIZE = getpagesize()

Procedure GetRSS()

	Protected rss = 0
	Protected fp = fopen("/proc/self/statm", "r")
	If fp = #Null
		ProcedureReturn -1
	EndIf

	Protected format.s = "%*s%ld"
	Protected *buff = AllocateMemory(StringByteLength(format, #PB_Ascii))
	PokeS(*buff, format, StringByteLength(format, #PB_Ascii), #PB_Ascii)

	Protected ret = fscanf(fp, *buff, @rss)
	FreeMemory(*buff)
	If ret <> 1
		fclose(fp)
		ProcedureReturn -2 
	EndIf
	fclose(fp)

	ProcedureReturn rss * SC_PAGESIZE / 1024 ; ko
EndProcedure


Procedure Thread(val)
	Protected mystruct1.Person
	Protected mystruct2.Person
	Protected mystruct3.Person
	Protected mystruct4.Person
	Protected mystruct5.Person

	Debug "rss="+GetRSS()

	Delay(10000)

EndProcedure


; main
Delay(2000)

Debug "first rss="+GetRSS()

For i = 1 To 20
	thid = CreateThread(@Thread(), 1)
	Delay(500)
Next
Debug "20 threads created"

WaitThread(thid)

Debug "last rss="+GetRSS()

I don't know if my example is the best to demonstrate what I mean.

Thanks.

[benubi]

You should use Protected keyword, especially in Threads (and enable ThreadSafe option). Perhaps this will fix some issues. I suspect the format.s and other variables to be put into shared memory and possibly released on thread exit. Meaning one thread overwriting the format.s of the other either with a new pointer or with zero when exiting. You can also use the Threaded keyword in place of Define's and Globals to avoid a few problems and have a comfortable setup, but you need ThreadSafe option to be on for it to work correctly.

[tatanas]

I always enable ThreadSafe option and use protected keyword (it was a omission in this sample, I corrected it).
Maybe it's just the way Linux handle the memory or displays it through RES...

[tored]

But how does the actual memory leak occur that you want to track? You use AllocateMemory but you are unsure if you use FreeMemory correctly?

[NicTheQuick]

Btw you can make your conversion to Ascii much easier like so:

Code: Select all

	*buff = Ascii(format)
	Protected ret = fscanf(fp, *buff, @rss)
	FreeMemory(*buff)

And it's even more simple if you use pseudotypes for fscanf:

Code: Select all

ImportC ""
	...
	fscanf(stream.i, format.p-ascii, size.i)
	...
EndImport
...
Procedure GetRSS()
	...
	Protected ret = fscanf(fp, format, @rss)
	...
EndProcedure
...

[tatanas]

But how does the actual memory leak occur that you want to track? You use AllocateMemory but you are unsure if you use FreeMemory correctly?

I don't know if this is a memory leak or the normal behaviour of the memory linux manager.
I checked the several AllocateMemory inside the treads and freed them.

In my example you can see the RES memory incresing and not decreasing after thread end. Maybe it's not a "leak" and my example is not representative enough.

NicTheQuick : thank you for your advices

[tored]

But how does the actual memory leak occur that you want to track? You use AllocateMemory but you are unsure if you use FreeMemory correctly?

I don't know if this is a memory leak or the normal behaviour of the memory linux manager.
I checked the several AllocateMemory inside the treads and freed them.

In my example you can see the RES memory incresing and not decreasing after thread end. Maybe it's not a "leak" and my example is not representative enough.

NicTheQuick : thank you for your advices

If you have not already seen this on the forum but the most common debugging technique for this is to overload AllocateMemory and FreeMemory with macros and store every allocation and free to detect any memory leak.

You can search the forum for multiple solutions, here is one that supports multiple threads.

viewtopic.php?f=12&t=39168

In this thread I have posted my module based variant, however not thread safe

viewtopic.php?t=79581

The general gist is to track every allocation in a list of structs that tracks pointer, file and line number and when freeing a pointer the entry in the list is removed (to make it thread safe the list needs to be protected by a mutex). On program exist the list should be empty otherwise you have a memory leak.

[Fred]

You can also use valgrind as you are on Linux. Just compile with -ds flag to have debug symbols and it should work.

[tatanas]

Thank you for your advices.

I'm trying Valgrind and no leak error for now.