HTTP(S) with WININET

roleg · Post by **roleg** » Fri Aug 17, 2012 10:41 am

I coded simple procedure for self usage.
This can GET,POST with SecureLayer,Cookies,User-Agent,Referer,Proxy,Timeout and can disable autoredirect.

Code: Select all

Procedure.s do_request(host.s, page.s="", post_data.s="",  cookie.s="", is_secure.b=#False, user_agent.s="", referer.s="", proxy.s="", timeout.l=1000, redirect.b=#True)
  If Not proxy="" : access_type.i=3 : Else : access_type.i=1 : EndIf
  open_handle = InternetOpen_(user_agent,access_type,proxy,"",0)
  InternetSetOption_(open_handle, 2,timeout,4)
  If is_secure
    port.i=443
    flag.l=$00800000|$00001000|$00002000|$00080000|$00000100|$04000000
  Else
    port.i=80
    flag.l=$00080000|$00000100|$04000000
  EndIf
  If Not redirect : flag|$00200000 : EndIf
  If Not post_data="" : verb.s="POST" : Else : verb.s="GET" : EndIf
  If page="" : page="/" : EndIf
  If user_agent="" : user_agent="Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" : EndIf
  connect_handle = InternetConnect_(open_handle,host,port,"","",3,0,0)
  request_handle = HttpOpenRequest_(connect_handle,verb,page,"",referer,0,flag,0)
  If verb="POST"
    headers.s = "Content-Type: application/x-www-form-urlencoded" +Chr(13)+Chr(10)
    HttpAddRequestHeaders_(request_handle,headers,Len(headers), $80000000|$20000000)
  EndIf
  If Not cookie=""
    headers.s = "Cookie: "+cookie+Chr(13)+Chr(10)
    HttpAddRequestHeaders_(request_handle,headers,Len(headers), $80000000|$20000000)
  EndIf
  send_handle = HttpSendRequest_(request_handle,"",0,post_data,Len(post_data))
  buffer.s = Space(1024)
  Repeat
    InternetReadFile_(request_handle,@buffer,1024,@bytes_read.l)
    result.s + Left(buffer,bytes_read)
    buffer = Space(1024)
  Until bytes_read=0
  InternetCloseHandle_(open_handle)
  InternetCloseHandle_(connect_handle)
  InternetCloseHandle_(request_handle)
  InternetCloseHandle_(send_handle)
  ProcedureReturn result
EndProcedure

Debug do_request("google.com");short call, only GET request for index file on http://google.com
Debug do_request("google.com","/xxx.php","her=vam","vam=her",#True,"Anonym browser 0.1","http://rambler.ru/ref.php","127.0.0.1:8888",2000,#False);full call,
;POST data with cookies,User-Agent:modiffed,referer,proxy,timeout,autoredirect:off to /xxx.php file on https://google.com

jesperbrannmark · Post by **jesperbrannmark** » Fri Aug 17, 2012 1:08 pm

Oh that is very nice! simple and nice.

jesperbrannmark · Post by **jesperbrannmark** » Fri Aug 17, 2012 1:13 pm

Spontanous look like it is allowing threading as well (compile threadsafe)

Code: Select all

Procedure.s do_request(thread.i,host.s, page.s="", post_data.s="",  cookie.s="", is_secure.b=#False, user_agent.s="", referer.s="", proxy.s="", timeout.l=1000, redirect.b=#True)
  Debug "THREAD STARTED:"+Str(thread)
  If Not proxy="" : access_type.i=3 : Else : access_type.i=1 : EndIf
  open_handle = InternetOpen_(user_agent,access_type,proxy,"",0)
  InternetSetOption_(open_handle, 2,timeout,4)
  If is_secure
    port.i=443
    flag.l=$00800000|$00001000|$00002000|$00080000|$00000100|$04000000
  Else
    port.i=80
    flag.l=$00080000|$00000100|$04000000
  EndIf
  If Not redirect : flag|$00200000 : EndIf
  If Not post_data="" : verb.s="POST" : Else : verb.s="GET" : EndIf
  If page="" : page="/" : EndIf
  If user_agent="" : user_agent="Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" : EndIf
  connect_handle = InternetConnect_(open_handle,host,port,"","",3,0,0)
  request_handle = HttpOpenRequest_(connect_handle,verb,page,"",referer,0,flag,0)
  If verb="POST"
    headers.s = "Content-Type: application/x-www-form-urlencoded" +Chr(13)+Chr(10)
    HttpAddRequestHeaders_(request_handle,headers,Len(headers), $80000000|$20000000)
  EndIf
  If Not cookie=""
    headers.s = "Cookie: "+cookie+Chr(13)+Chr(10)
    HttpAddRequestHeaders_(request_handle,headers,Len(headers), $80000000|$20000000)
  EndIf
  send_handle = HttpSendRequest_(request_handle,"",0,post_data,Len(post_data))
  buffer.s = Space(1024)
  Repeat
    InternetReadFile_(request_handle,@buffer,1024,@bytes_read.l)
    result.s + Left(buffer,bytes_read)
    buffer = Space(1024)
  Until bytes_read=0
  InternetCloseHandle_(open_handle)
  InternetCloseHandle_(connect_handle)
  InternetCloseHandle_(request_handle)
  InternetCloseHandle_(send_handle)
  Debug "THREAD DONE "+Str(thread)
  ProcedureReturn result
EndProcedure
Procedure req(dummy.i)
  Debug do_request(dummy.i,"google.com","","","",#True);short call, only GET request for index file on http://google.comen
EndProcedure
OpenWindow(0,100,100,640,480,"")
Debug "START"
For i=1 To 50
  CreateThread(@req(),i)
Next
Debug "DONE"
Repeat
Until WaitWindowEvent()=#PB_Event_CloseWindow
;Debug do_request("google.com","/xxx.php","her=vam","vam=her",#True,"Anonym browser 0.1","http://rambler.ru/ref.php","127.0.0.1:8888",2000,#False);full call,
;POST data with cookies,User-Agent:modiffed,referer,proxy,timeout,autoredirect:off to /xxx.php file on https://google.com

roleg · Post by **roleg** » Sat Aug 18, 2012 8:21 am

jesperbrannmark, yes
I use it in my SQL vulnerability scanner, about 40 threads without any problem.

roleg · Post by **roleg** » Tue Mar 26, 2013 2:55 pm

there is a some problem
randomly, after 4-7 hours of work in multithreaded mode
we get a memory error

20 threads
safe threads: ON

tested:
XP, Core2Duo 3 Ghz, 2 Gb
XP, Core i7 3 Ghz, 4 Gb

jesperbrannmark · Post by **jesperbrannmark** » Tue Mar 26, 2013 3:20 pm

What PB version are you using? And do you notice anything on mem/cpu usage along the way?

roleg · Post by **roleg** » Tue Mar 26, 2013 3:30 pm

PB 5.0
CPU: 10-25%
RAM: 100-200 Mb

jesperbrannmark · Post by **jesperbrannmark** » Tue Mar 26, 2013 3:54 pm

Is the ram constant or does it "grow by the hour"?

roleg · Post by **roleg** » Wed Mar 27, 2013 7:50 am

The number of sites checked grows, filling Map() + content sites temporarily loaded into RAM, and expands by about 100 Mb constantly

P.S. I add small timeout Delay(100) after CloseHandle and it was less falling out with an error.
it seems that the handles do not have time to close and to free

8 hours, normal flight...

infratec · Post by **infratec** » Wed Aug 28, 2013 7:41 am

Hi,

since I needed it:

I implemented that an invalid (homemade) certificate is tolerated.

Code: Select all

; http://support.microsoft.com/kb/q182888

EnableExplicit

#INTERNET_DEFAULT_HTTP_PORT = 80
#INTERNET_DEFAULT_HTTPS_PORT = 443

#INTERNET_OPEN_TYPE_DIRECT = 1
#INTERNET_OPEN_TYPE_PROXY = 3

#INTERNET_SERVICE_HTTP = 3

#INTERNET_FLAG_PRAGMA_NOCACHE =           $00000100
#INTERNET_FLAG_IGNORE_CERT_CN_INVALID =   $00001000
#INTERNET_FLAG_IGNORE_CERT_DATE_INVALID = $00002000
#INTERNET_FLAG_NO_COOKIES =               $00080000
#INTERNET_FLAG_NO_AUTO_REDIRECT =         $00200000
#INTERNET_FLAG_SECURE =                   $00800000
#INTERNET_FLAG_DONT_CACHE =               $04000000


#HTTP_ADDREQ_FLAG_ADD =     $20000000
#HTTP_ADDREQ_FLAG_REPLACE = $80000000


#ERROR_INTERNET_INVALID_CA = 12045

#INTERNET_OPTION_SECURITY_FLAGS = 31

#SECURITY_FLAG_IGNORE_UNKNOWN_CA = $00000100



Procedure.s do_request(host.s, page.s="", post_data.s="",  cookie.s="", is_secure.i=#False, user_agent.s="", referer.s="", proxy.s="", timeout.l=1000, redirect.i=#True)
  
  Protected.l flags, bytes_read, dwFlags, dwBuffLen
  Protected.i open_handle, port, connect_handle, request_handle, send_handle, Ok, access_type, LastError
  Protected.s verb, headers, buffer, result
  
  
  If proxy = "" : access_type =  #INTERNET_OPEN_TYPE_DIRECT : Else : access_type = #INTERNET_OPEN_TYPE_PROXY : EndIf
  If user_agent = "" : user_agent = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" : EndIf
  open_handle = InternetOpen_(user_agent, access_type, "", "", 0)
  
  InternetSetOption_(open_handle, 2, timeout, 4)
  
  flags = #INTERNET_FLAG_PRAGMA_NOCACHE
  flags | #INTERNET_FLAG_NO_COOKIES
  flags | #INTERNET_FLAG_DONT_CACHE
  
  If is_secure
    port = #INTERNET_DEFAULT_HTTPS_PORT
    
    flags | #INTERNET_FLAG_SECURE
    flags | #INTERNET_FLAG_IGNORE_CERT_CN_INVALID
    flags | #INTERNET_FLAG_IGNORE_CERT_DATE_INVALID
  Else
    port = #INTERNET_DEFAULT_HTTP_PORT
  EndIf
  
  Debug host
  connect_handle = InternetConnect_(open_handle, host, port, "", "", #INTERNET_SERVICE_HTTP, 0, 0)
  ;Debug connect_handle
  
  
  If Not redirect : flags | #INTERNET_FLAG_NO_AUTO_REDIRECT : EndIf
  If post_data = "" : verb = "GET" : Else : verb = "POST" : EndIf
  Debug verb
  If page = "" : page = "/" : EndIf
  Debug page
  request_handle = HttpOpenRequest_(connect_handle, verb, page, "", referer, #Null, flags, 0)
  ;Debug request_handle
  
  If verb = "POST"
    ;headers = "Content-Type: application/x-www-form-urlencoded" + #CRLF$
    headers = "Content-Type: text/html" + #CRLF$
    HttpAddRequestHeaders_(request_handle, headers, Len(headers), #HTTP_ADDREQ_FLAG_ADD | #HTTP_ADDREQ_FLAG_REPLACE)
  EndIf
  If cookie <> ""
    headers = "Cookie: "+ cookie + #CRLF$
    HttpAddRequestHeaders_(request_handle, headers, Len(headers), #HTTP_ADDREQ_FLAG_ADD | #HTTP_ADDREQ_FLAG_REPLACE)
  EndIf
  
  
  Ok = 0
  Repeat
    send_handle = HttpSendRequest_(request_handle, #Null, 0, post_data, Len(post_data))
    
    If send_handle = 0
      LastError = GetLastError_()
      
      Debug "Error " + Str(LastError)
      
      If LastError = #ERROR_INTERNET_INVALID_CA
        
        dwBuffLen = SizeOf(dwFlags)
        
        InternetQueryOption_(request_handle, #INTERNET_OPTION_SECURITY_FLAGS, @dwFlags, @dwBuffLen)
        
        dwFlags | #SECURITY_FLAG_IGNORE_UNKNOWN_CA
        InternetSetOption_(request_handle, #INTERNET_OPTION_SECURITY_FLAGS, @dwFlags, SizeOf(dwFlags))
        Ok + 1
      Else
        Ok = 2
      EndIf
    Else
      Ok = 2
    EndIf
  Until Ok = 2
  
  
  buffer = Space(1024)
  Repeat
    InternetReadFile_(request_handle, @buffer, 1024, @bytes_read)
    result + Left(buffer, bytes_read)
    buffer = Space(1024)
  Until bytes_read = 0
  
  InternetCloseHandle_(open_handle)
  InternetCloseHandle_(connect_handle)
  InternetCloseHandle_(request_handle)
  InternetCloseHandle_(send_handle)
  
  ProcedureReturn result
  
EndProcedure

Bernd

ricardo · Post by **ricardo** » Mon Feb 17, 2014 9:50 pm

This fails for me as soon as i add the proxy, why? The proxy works fine (tested with ScrapeBox)

Code: Select all

Debug do_request("google.com","","","",#True,"","","182.253.48.166:80",2000)

ostapas · Post by **ostapas** » Mon Feb 17, 2014 10:41 pm

I've slightly modified this procedure for my needs(return cookies, download/upload file, etc.) http://pastebin.com/sh6qZq8m Didn't investigate where the problem is, but it seems to work with the version I posted, e.g.

Code: Select all

Debug HttpRequest("https://google.com", 1, "", "", "", "", "", "195.171.194.91:80")

However, TimeOut parameter does not work when using a proxy(seems to be a MS bug)

ricardo · Post by **ricardo** » Thu Mar 20, 2014 5:12 pm

ostapas wrote:I've slightly modified this procedure for my needs(return cookies, download/upload file, etc.) http://pastebin.com/sh6qZq8m Didn't investigate where the problem is, but it seems to work with the version I posted, e.g.
Code: Select all
Debug HttpRequest("https://google.com", 1, "", "", "", "", "", "195.171.194.91:80")
However, TimeOut parameter does not work when using a proxy(seems to be a MS bug)

Works in Google, but not with all sites, why? I will test more.
This slightly modification is great indeed.

coco2 · Post by **coco2** » Fri Mar 21, 2014 12:53 pm

What sort of encoding does it return the data in?

PureBasic Forums - English

HTTP(S) with WININET

HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET

Re: HTTP(S) with WININET