Page 1 of 3
Threadsafe Libcurl HTTPS Post
Posted: Sun Jul 02, 2017 6:42 pm
by Liqu
I use this lib and example and got random crashes after few seconds,
how to make it threadsafe? I want to use it for stress-testing my api server, thank you.
Code: Select all
[00:36:12] Waiting for executable to start...
[00:36:12] Executable type: Windows - x86 (32bit, Unicode, Thread, Purifier)
[00:36:12] Executable started.
[00:36:14] [ERROR] Line: 17
[00:36:14] [ERROR] Overflow in a string memory block.
[00:37:02] Waiting for executable to start...
[00:37:02] Executable type: Windows - x86 (32bit, Unicode, Thread, Purifier)
[00:37:02] Executable started.
[00:37:06] [ERROR] Line: 29
[00:37:06] [ERROR] Overflow in a string memory block.
[00:40:09] Executable type: Windows - x86 (32bit, Unicode, Thread, Purifier)
[00:40:09] Executable started.
[00:40:17] [ERROR] pb_curl.pbi (Line: 1045)
[00:40:17] [ERROR] Invalid memory access. (read error at address 4)
The codes:
https://github.com/deseven/pbsamples/tr ... rm/libcurl
Code: Select all
IncludeFile "pb_curl.pbi"
; working with static libcurl
InitNetwork()
Procedure liqu_post(tread)
; Debug "Thread on: "+Str(tread)
Protected curl = curl_easy_init()
Protected url.s = str2curl("http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1")
Protected agent.s = str2curl("pbcurl/1.0")
; cookie.s = str2curl("var=value;")
; post.s = str2curl("login=mylogin&password=mypassword")
Protected header.s = str2curl("Cache-Control: no-cache")
Protected time_start, res, time_end, resData.s
If curl
curl_easy_setopt(curl,#CURLOPT_URL,@url)
curl_easy_setopt(curl,#CURLOPT_IPRESOLVE,#CURL_IPRESOLVE_V4)
; curl_easy_setopt(curl,#CURLOPT_COOKIE,@cookie)
; curl_easy_setopt(curl,#CURLOPT_POSTFIELDS,@post)
curl_easy_setopt(curl,#CURLOPT_USERAGENT,@agent)
curl_easy_setopt(curl,#CURLOPT_TIMEOUT,30)
curl_easy_setopt(curl,#CURLOPT_FOLLOWLOCATION,1)
*header = curl_slist_append(0,header)
curl_easy_setopt(curl,#CURLOPT_HTTPHEADER,*header)
curl_easy_setopt(curl,#CURLOPT_WRITEFUNCTION,@curlWriteData())
time_start = ElapsedMilliseconds()
res = curl_easy_perform(curl)
time_end = ElapsedMilliseconds() - time_start
resData = curlGetData()
curl_easy_getinfo(curl,#CURLINFO_RESPONSE_CODE,@resHTTP)
Debug Str(tread)+" HTTP result: " + Str(res)
If Not res
Debug Str(tread)+" : "+Str(time_end) + " ms"
Debug Str(tread)+" HTTP code: " + Str(resHTTP)
Debug Str(tread)+" HTTP data: " + #CRLF$ + resData
EndIf
curl_easy_cleanup(curl)
Else
Debug Str(tread)+" can't init curl!"
EndIf
EndProcedure
Procedure liqu_repeat_post(thread)
Repeat
liqu_post(thread)
Delay(100)
ForEver
EndProcedure
Repeat
i=i+1
CreateThread(@liqu_repeat_post(), i)
Delay(20)
Until i>100
Repeat
Delay(500000)
ForEver
Re: Threadsafe Libcurl HTTPS Post
Posted: Sun Jul 02, 2017 8:42 pm
by infratec
Hi,
if use curl_slist_append() then you need to free it with
It is not released with curl_easy_cleanup(curl)
Also
Code: Select all
Protected url.s = str2curl("http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1")
is wrong.
str2curl returns a pointer to a static variable.
You should use it like:
Code: Select all
Protected url.s = "http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1"
curl_easy_setopt(curl, #CURLOPT_URL, str2curl(url))
Bernd
Re: Threadsafe Libcurl HTTPS Post
Posted: Sun Jul 02, 2017 9:05 pm
by Olliv
Hello,
to grow ways of find some solutions, add a describing of lines 17, 29 and 1045.
(edit: ups did not seen message of infratec, excuse)
Re: Threadsafe Libcurl HTTPS Post
Posted: Mon Jul 03, 2017 4:49 am
by Liqu
Hi infratec and Olliv,
I still got the error after few seconds,
Code: Select all
[01:33:32] Waiting for executable to start...
[01:33:32] Executable type: Windows - x86 (32bit, Unicode)
[01:33:32] Executable started.
[01:33:34] [ERROR] Line: 31
[01:33:34] [ERROR] Invalid memory access. (write error at address 0)
[01:33:34] [ERROR] Line: 31
[01:33:34] [ERROR] Invalid memory access. (write error at address 208904192)
[01:33:34] The Program execution has finished.
[01:33:53] Waiting for executable to start...
[01:33:53] Executable type: Windows - x86 (32bit, Unicode, Thread, Purifier)
[01:33:54] Executable started.
[01:33:57] [ERROR] pb_curl.pbi (Line: 1046)
[01:33:57] [ERROR] Invalid memory access. (write error at address 4)
[01:34:06] The Program was killed.
[10:00:43] Executable type: Windows - x86 (32bit, Unicode, Thread, Purifier)
[10:00:43] Executable started.
[10:00:45] [ERROR] pb_curl.pbi (Line: 1046)
[10:00:45] [ERROR] Invalid memory access. (write error at address 4)
[10:44:21] The Program was killed.
[10:44:46] Executable type: Windows - x86 (32bit, Unicode, Thread, Purifier)
[10:44:46] Executable started.
[10:44:48] [ERROR] pb_curl.pbi (Line: 1046)
[10:44:48] [ERROR] Invalid memory access. (write error at address 4)
[10:45:58] Waiting for executable to start...
[10:45:58] Executable type: Windows - x86 (32bit, Unicode, Thread, Purifier)
[10:45:59] Executable started.
[10:46:02] [ERROR] Line: 31
[10:46:02] [ERROR] Overflow in a string memory block.
[10:48:11] Executable type: Windows - x86 (32bit, Unicode, Thread, Purifier)
[10:48:11] Executable started.
[10:48:13] [ERROR] pb_curl.pbi (Line: 1045)
[10:48:13] [ERROR] Invalid memory access. (read error at address 4)
Code: Select all
Line: 31
res = curl_easy_perform(curl)
Line 1045 - 1046:
ReturnData.s = ReceivedData.s
ReceivedData.s = ""
here's the updated code:
https://github.com/deseven/pbsamples/tr ... rm/libcurl
Code: Select all
IncludeFile "pb_curl.pbi"
; working with static libcurl
InitNetwork()
Procedure liqu_post(tread)
; Debug "Thread on: "+Str(tread)
Protected curl = curl_easy_init()
Protected url.s = "http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1"
Protected agent.s = str2curl("pbcurl/1.0")
; cookie.s = str2curl("var=value;")
; post.s = str2curl("login=mylogin&password=mypassword")
Protected header.s = str2curl("Cache-Control: no-cache")
Protected time_start, res, time_end, resData.s
If curl
curl_easy_setopt(curl,#CURLOPT_URL, str2curl(url))
curl_easy_setopt(curl,#CURLOPT_IPRESOLVE,#CURL_IPRESOLVE_V4)
; curl_easy_setopt(curl,#CURLOPT_COOKIE,@cookie)
; curl_easy_setopt(curl,#CURLOPT_POSTFIELDS,@post)
curl_easy_setopt(curl,#CURLOPT_USERAGENT,@agent)
curl_easy_setopt(curl,#CURLOPT_TIMEOUT,30)
curl_easy_setopt(curl,#CURLOPT_FOLLOWLOCATION,1)
*header = curl_slist_append(0,header)
curl_easy_setopt(curl,#CURLOPT_HTTPHEADER,*header)
curl_easy_setopt(curl,#CURLOPT_WRITEFUNCTION,@curlWriteData())
time_start = ElapsedMilliseconds()
res = curl_easy_perform(curl)
time_end = ElapsedMilliseconds() - time_start
resData = curlGetData()
curl_easy_getinfo(curl,#CURLINFO_RESPONSE_CODE,@resHTTP)
Debug Str(tread)+" HTTP result: " + Str(res)
If Not res
Debug Str(tread)+" : "+Str(time_end) + " ms"
Debug Str(tread)+" HTTP code: " + Str(resHTTP)
Debug Str(tread)+" HTTP data: " + #CRLF$ + resData
EndIf
curl_slist_free_all(*header)
curl_easy_cleanup(curl)
Else
Debug Str(tread)+" can't init curl!"
EndIf
EndProcedure
Procedure liqu_repeat_post(thread)
Repeat
liqu_post(thread)
Delay(100)
ForEver
EndProcedure
Repeat
i=i+1
CreateThread(@liqu_repeat_post(), i)
Delay(10)
Until i>100
Repeat
Delay(500000)
ForEver
Thank you
Re: Threadsafe Libcurl HTTPS Post
Posted: Mon Jul 03, 2017 7:18 am
by djes
Is your program working with only one thread ? All the string pointers should be verified. And don't forget the use of threaded variables, see «threaded» in the doc.
Re: Threadsafe Libcurl HTTPS Post
Posted: Mon Jul 03, 2017 7:41 am
by infratec
Hi,
the custom write function in libcurl.pbi is not threadsave.
You need your own.
Code: Select all
CompilerIf Not #PB_Compiler_Thread
CompilerError "Enable Thread Save in compiler options!"
CompilerEndIf
EnableExplicit
Define ReceivedData.s ; not used, but libcurl.pbi need it
Structure ThreadStructure
No.i
ID.i
response_code.l
ReceivedData.s
Duartion.i
Finished.i
EndStructure
NewList ThreadList.ThreadStructure()
IncludeFile "libcurl.pbi"
ProcedureC curlOwnWriteData(*ptr, Size, NMemB, *userData.ThreadStructure)
Protected SizeProper.i = Size & 255
Protected NMemBProper.i = NMemB
*userData\ReceivedData + PeekS(*ptr, SizeProper * NMemBProper, #PB_UTF8|#PB_ByteLength)
ProcedureReturn SizeProper * NMemBProper
EndProcedure
Procedure liqu_post(*Parameter.ThreadStructure)
; Debug "Thread on: "+Str(Thread)
Protected.i curl, time_start, res, time_end
Protected.s url, agent, header
Protected *header
curl = curl_easy_init()
If curl
url = "http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1"
curl_easy_setopt(curl,#CURLOPT_URL, str2curl(url))
curl_easy_setopt(curl,#CURLOPT_IPRESOLVE, #CURL_IPRESOLVE_V4)
agent = "pbcurl/1.0"
curl_easy_setopt(curl,#CURLOPT_USERAGENT, str2curl(agent))
curl_easy_setopt(curl,#CURLOPT_TIMEOUT,30)
curl_easy_setopt(curl,#CURLOPT_FOLLOWLOCATION,1)
header = "Cache-Control: no-cache"
*header = curl_slist_append(0, header)
curl_easy_setopt(curl,#CURLOPT_HTTPHEADER, *header)
curl_easy_setopt(curl, #CURLOPT_WRITEDATA, *Parameter)
curl_easy_setopt(curl, #CURLOPT_WRITEFUNCTION, @curlOwnWriteData())
time_start = ElapsedMilliseconds()
res = curl_easy_perform(curl)
*Parameter\Duartion = ElapsedMilliseconds() - time_start
If res = #CURLE_OK
curl_easy_getinfo(curl, #CURLINFO_RESPONSE_CODE, @*Parameter\response_code)
EndIf
curl_slist_free_all(*header)
curl_easy_cleanup(curl)
Else
Debug Str(*Parameter\No)+" can't init curl!"
EndIf
*Parameter\Finished = #True
EndProcedure
Define *ElementPtr, i.i, Finished.i
InitNetwork()
Repeat
i + 1
*ElementPtr = AddElement(ThreadList())
ThreadList()\No = i
ThreadList()\ID = CreateThread(@liqu_post(), *ElementPtr)
Until i > 99
Repeat
Finished = #True
ForEach ThreadList()
If Not ThreadList()\Finished
Finished = #False
Break
EndIf
Next
Until Finished
ForEach ThreadList()
Debug Str(ThreadList()\No) + " : "+Str(ThreadList()\Duartion) + " ms"
Debug Str(ThreadList()\No)+" HTTP code: " + Str(ThreadList()\response_code)
Debug Str(ThreadList()\No)+" HTTP data: " + #CRLF$ + ThreadList()\ReceivedData
Next
Bernd
Re: Threadsafe Libcurl HTTPS Post
Posted: Mon Jul 03, 2017 10:42 am
by Bisonte
@infratec
This looks wrong :
Code: Select all
*header = curl_slist_append(0, header)
Must it be
Code: Select all
*header = curl_slist_append(curl, header)
?
Re: Threadsafe Libcurl HTTPS Post
Posted: Mon Jul 03, 2017 10:50 am
by infratec
Hi,
no, it's not wrong. It only looks strange.
Normally I would call it with:
Code: Select all
*header = curl_slist_append(*header, "Cache-Control: no-cache")
But since *header is NULL at the first call you can call it also with
Code: Select all
*header = curl_slist_append(#Null, header)
(For the first call)
The functions extends the list of appends.
You supply the old *buffer you received from the function and get the new extended *buffer back.
(If you call it more than once)
The list itself has nothing to do with the curl handle.
Later you provide the list to the curl handle with:
Code: Select all
curl_easy_setopt(curl,#CURLOPT_HTTPHEADER, *header)
Bernd
Re: Threadsafe Libcurl HTTPS Post
Posted: Mon Jul 03, 2017 11:07 am
by Bisonte
ah ok. Thx for explaining it.
Re: Threadsafe Libcurl HTTPS Post
Posted: Mon Jul 03, 2017 11:46 am
by djes
I remember that I had problem with the libcurl.pbi's str2curl() function too. It was flawed for my usage... Someone could take a look? Here's the one that I've used for PBMap.
Code: Select all
Procedure.s str2curl(string.s)
Protected *curlstring, newstring.s
*curlstring = AllocateMemory(Len(string) + 2) ;corrected from 1 to 2 on infratec's advice
If *curlstring
PokeS(*curlstring, string, -1, #PB_Ascii)
newstring = PeekS(*curlstring, -1)
FreeMemory(*curlstring)
ProcedureReturn newstring
Else
ProcedureReturn ""
EndIf
EndProcedure
Re: Threadsafe Libcurl HTTPS Post
Posted: Mon Jul 03, 2017 1:42 pm
by Bisonte
I use the UTF8() or ASCII() function of PB for it, and free it after use like this
Code: Select all
If User_Agent <> ""
*Agent = UTF8(User_Agent)
curl_easy_setopt(*curl, #CURLOPT_USERAGENT, *Agent)
FreeMemory(*Agent)
EndIf
Because, I do not understand the original str2curl() from deseven.
With my logic, there is in the original function a memory leak.
Re: Threadsafe Libcurl HTTPS Post
Posted: Mon Jul 03, 2017 2:12 pm
by infratec
Hi,
my current version + some Macros
Code: Select all
CompilerIf #PB_Compiler_OS = #PB_OS_Linux
curl_version_() As "curl_version"
CompilerElse
curl_version_() As "_curl_version"
CompilerEndIf
CompilerIf #PB_Compiler_OS = #PB_OS_Linux
curl_easy_strerror_(errornum.i) As "curl_easy_strerror"
CompilerElse
curl_easy_strerror_(errornum.i) As "_curl_easy_strerror"
CompilerEndIf
Code: Select all
Procedure.i str2curl(string.s)
Static *curlstring
If *curlstring : FreeMemory(*curlstring) : EndIf
*curlstring = UTF8(string)
ProcedureReturn *curlstring
EndProcedure
Macro curl_easy_strerror(error)
PeekS(curl_easy_strerror_(error), -1, #PB_UTF8)
EndMacro
Macro curl_version()
PeekS(curl_version_(), -1, #PB_UTF8)
EndMacro
Bernd
Re: Threadsafe Libcurl HTTPS Post
Posted: Mon Jul 03, 2017 2:30 pm
by djes
Thank you both

Re: Threadsafe Libcurl HTTPS Post
Posted: Wed Jul 05, 2017 1:13 pm
by Liqu
Hi infratec,
I want to simulate 1000 concurrent user for a set amount of time so i modified your code, but i got this error:
Code: Select all
[19:08:44] The Program was killed.
[19:08:48] Waiting for executable to start...
[19:08:47] Executable type: Windows - x86 (32bit, Unicode, Thread, Purifier)
[19:08:48] Executable started.
[19:10:04] [ERROR] Line: 31
[19:10:04] [ERROR] Invalid memory access. (write error at address 355552)
Line 31 :
Code: Select all
*userData\ReceivedData + PeekS(*ptr, SizeProper * NMemBProper, #PB_UTF8|#PB_ByteLength)
here's my code:
Code: Select all
CompilerIf Not #PB_Compiler_Thread
CompilerError "Enable Thread Save in compiler options!"
CompilerEndIf
EnableExplicit
Define ReceivedData.s ; not used, but libcurl.pbi need it
Structure ThreadStructure
No.i
ID.i
response_code.l
ReceivedData.s
Duartion.i
Finished.i
EndStructure
Threaded NewList ThreadList.ThreadStructure()
IncludeFile "pb_curl.pbi"
Threaded SizeProper.i, NMemBProper.i, *ptr, Size, NMemB
ProcedureC curlOwnWriteData(*ptr, Size, NMemB, *userData.ThreadStructure)
Protected SizeProper.i = Size & 255
Protected NMemBProper.i = NMemB
*userData\ReceivedData + PeekS(*ptr, SizeProper * NMemBProper, #PB_UTF8|#PB_ByteLength)
ProcedureReturn SizeProper * NMemBProper
EndProcedure
Procedure liqu_post(*Parameter.ThreadStructure)
; Debug "Thread on: "+Str(Thread)
Protected.i curl, time_start, res, time_end
Protected.s url, agent, header
Protected *header
curl = curl_easy_init()
If curl
url = "http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1"
curl_easy_setopt(curl,#CURLOPT_URL, str2curl(url))
curl_easy_setopt(curl,#CURLOPT_IPRESOLVE, #CURL_IPRESOLVE_V4)
agent = "pbcurl/1.0"
curl_easy_setopt(curl,#CURLOPT_USERAGENT, str2curl(agent))
curl_easy_setopt(curl,#CURLOPT_TIMEOUT,30)
curl_easy_setopt(curl,#CURLOPT_FOLLOWLOCATION,1)
header = "Cache-Control: no-cache"
*header = curl_slist_append(0, header)
curl_easy_setopt(curl,#CURLOPT_HTTPHEADER, *header)
curl_easy_setopt(curl, #CURLOPT_WRITEDATA, *Parameter)
curl_easy_setopt(curl, #CURLOPT_WRITEFUNCTION, @curlOwnWriteData())
time_start = ElapsedMilliseconds()
res = curl_easy_perform(curl)
*Parameter\Duartion = ElapsedMilliseconds() - time_start
; Debug Str(*Parameter\No)+" "+ Str(*Parameter\Duartion)
If res = #CURLE_OK
curl_easy_getinfo(curl, #CURLINFO_RESPONSE_CODE, @*Parameter\response_code)
EndIf
curl_slist_free_all(*header)
curl_easy_cleanup(curl)
Else
Debug Str(*Parameter\No)+" can't init curl!"
EndIf
*Parameter\Finished = #True
EndProcedure
Threaded *ElementPtr.ThreadStructure, *Parameter.ThreadStructure, i.i, Finished.i
InitNetwork()
Procedure liqu_repeat_post(*Parameter.ThreadStructure)
Repeat
liqu_post(*Parameter)
Delay(1000)
ForEver
EndProcedure
Repeat
i + 1
*ElementPtr = AddElement(ThreadList())
ThreadList()\No = i
ThreadList()\ID = CreateThread(@liqu_repeat_post(), *ElementPtr)
Debug "Thread Started: "+Str(i)
Delay(1)
Until i > 1000
Repeat
Delay(1000)
ForEver
; Repeat
; Finished = #True
; ForEach ThreadList()
; If Not ThreadList()\Finished
; Finished = #False
; Break
; EndIf
; Next
; Until Finished
;
; ForEach ThreadList()
; Debug Str(ThreadList()\No) + " : "+Str(ThreadList()\Duartion) + " ms"
; Debug Str(ThreadList()\No)+" HTTP code: " + Str(ThreadList()\response_code)
; Debug Str(ThreadList()\No)+" HTTP data: " + #CRLF$ + ThreadList()\ReceivedData
; Next
and where to put this?
i tried to put this on curl.pbi and the app script but got error
infratec wrote:Hi,
my current version + some Macros
Code: Select all
CompilerIf #PB_Compiler_OS = #PB_OS_Linux
curl_version_() As "curl_version"
CompilerElse
curl_version_() As "_curl_version"
CompilerEndIf
CompilerIf #PB_Compiler_OS = #PB_OS_Linux
curl_easy_strerror_(errornum.i) As "curl_easy_strerror"
CompilerElse
curl_easy_strerror_(errornum.i) As "_curl_easy_strerror"
CompilerEndIf
Code: Select all
Procedure.i str2curl(string.s)
Static *curlstring
If *curlstring : FreeMemory(*curlstring) : EndIf
*curlstring = UTF8(string)
ProcedureReturn *curlstring
EndProcedure
Macro curl_easy_strerror(error)
PeekS(curl_easy_strerror_(error), -1, #PB_UTF8)
EndMacro
Macro curl_version()
PeekS(curl_version_(), -1, #PB_UTF8)
EndMacro
Bernd
thank you very much for the help
Re: Threadsafe Libcurl HTTPS Post
Posted: Wed Jul 05, 2017 1:57 pm
by infratec
Hi,
is also not thread safe.
Use this in your code:
Code: Select all
Procedure liqu_post(*Parameter.ThreadStructure)
; Debug "Thread on: "+Str(Thread)
Protected.i curl, time_start, res, time_end
Protected.s agent, header
Protected *header, *UTF8
curl = curl_easy_init()
If curl
*UTF8 = UTF8("http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1")
curl_easy_setopt(curl,#CURLOPT_URL, *UTF8)
FreeMemory(*UTF8)
curl_easy_setopt(curl,#CURLOPT_IPRESOLVE, #CURL_IPRESOLVE_V4)
*UTF8 = UTF8("pbcurl/1.0")
curl_easy_setopt(curl,#CURLOPT_USERAGENT, *UTF8)
FreeMemory(*UTF8)
curl_easy_setopt(curl,#CURLOPT_TIMEOUT,30)
curl_easy_setopt(curl,#CURLOPT_FOLLOWLOCATION,1)
header = "Cache-Control: no-cache"
*header = curl_slist_append(0, header)
curl_easy_setopt(curl,#CURLOPT_HTTPHEADER, *header)
curl_easy_setopt(curl, #CURLOPT_WRITEDATA, *Parameter)
curl_easy_setopt(curl, #CURLOPT_WRITEFUNCTION, @curlOwnWriteData())
time_start = ElapsedMilliseconds()
res = curl_easy_perform(curl)
*Parameter\Duartion = ElapsedMilliseconds() - time_start
; Debug Str(*Parameter\No)+" "+ Str(*Parameter\Duartion)
If res = #CURLE_OK
curl_easy_getinfo(curl, #CURLINFO_RESPONSE_CODE, @*Parameter\response_code)
EndIf
curl_slist_free_all(*header)
curl_easy_cleanup(curl)
Else
Debug Str(*Parameter\No)+" can't init curl!"
EndIf
*Parameter\Finished = #True
EndProcedure
And don't use stuff which you not know how to use
You don't need Threaded.
The first code snippets replaces already available stuff in libcurl.pbi inside the Import section.
str2curl is clear (but of course of the static variable not thread safe)
And the Macros uses adds the replaced Imports to make live in PB easier.
Bernd
Bernd