Proper HTTP GET&POST(cross-platform,chunking etc..)

Share your advanced PureBasic knowledge/code with the community.
tj1010
Enthusiast
Enthusiast
Posts: 716
Joined: Mon Feb 25, 2013 5:51 pm

Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by tj1010 »

I really needed stable cross-platform HTTP procedures I could just drop in source code, and could only find short unstable examples here and in archives.. I wrote this today and tested it with chunking&non-chunking HTTP POST&GET servers. Call them with Location header value for 301 and 302 responses.

If you want SSL and/or compression I suggest API, although you have to use bloated COM for flexible POST on Windows.

Let me know if you hit issues

Code: Select all

;Improvements to be made:
;-Handle SendNetworkData limit
;-Handle 301 and 302 transparantly by parsing and calling selves
;-Better default timeout value?
EnableExplicit
InitNetwork()

Procedure.s GetHeader(Packet$,Name$)
  ProcedureReturn Mid(Packet$,FindString(Packet$,Name$)+Len(Name$)+2,FindString(Packet$,#CRLF$,FindString(Packet$,Name$))-(FindString(Packet$,Name$)+Len(Name$)+2))
EndProcedure

Procedure.s HTTPPost(server$,port.l,vars$,headers$,timeout.l=6000)
  ;headers$ should already have #CRLF$ after each
  ;variables should already be encoded in server$ and vars$
  ;chunked responses are transparent, PB doesn't show 0-length packet
  Protected con.l
  Protected head$
  Protected part$
  Protected ret.l
  Protected bytes.l
  Protected timer.l
  Protected buffer$
  Protected response$
  con=OpenNetworkConnection(GetURLPart(server$,#PB_URL_Site),port,#PB_Network_TCP,timeout)
  If con
    ;build header
    head$="POST /"+GetURLPart(server$,#PB_URL_Path)
    If Len(GetURLPart(server$,#PB_URL_Parameters))>0 : head$=head$+"?"+GetURLPart(server$,#PB_URL_Parameters) : EndIf
    head$=head$+" HTTP/1.1"+#CRLF$
    If Len(headers$)>0 : head$=head$+headers$ : EndIf
    head$=head$+"Host: "+GetURLPart(server$,#PB_URL_Site)+#CRLF$
    head$=head$+"Accept: */*"+#CRLF$
    If CountString(headers$,"Content-Type:")=0 : head$=head$+"Content-Type: application/x-www-form-urlencoded"+#CRLF$ : EndIf
    head$=head$+"Connection: keep-alive"+#CRLF$
    head$=head$+"Content-Length: "+Str(Len(vars$))+#CRLF$
    head$=head$+#CRLF$
    ;send header
    timer=ElapsedMilliseconds()
    Repeat
      If ret>0
        ;send remaining
        part$=Mid(head$,ret,Len(head$))
        bytes=SendNetworkData(con,@part$,Len(part$))
      Else
        ;start sending
        bytes=SendNetworkData(con,@head$,Len(head$))
      EndIf
      If bytes<>-1 : ret=ret+bytes : EndIf
      If ElapsedMilliseconds()-timer>=timeout : CloseNetworkConnection(con) : ProcedureReturn "" : EndIf
    Until ret=Len(head$)
    ret=0
    ;send content
    timer=ElapsedMilliseconds()
    Repeat
      If ret>0
        ;send remaining
        part$=Mid(vars$,ret,Len(vars$))
        bytes=SendNetworkData(con,@part$,Len(part$))
      Else
        ;start sending
        bytes=SendNetworkData(con,@vars$,Len(vars$))
      EndIf
      If bytes<>-1 : ret=ret+bytes : EndIf
      If ElapsedMilliseconds()-timer>=timeout : CloseNetworkConnection(con) : ProcedureReturn "" : EndIf
    Until ret=Len(vars$)
    ret=0
    ;receive response
    timer=ElapsedMilliseconds()
    Repeat
      Delay(100)
      Select NetworkClientEvent(con)
        Case #PB_NetworkEvent_Data
          buffer$=Space(14500)
          bytes=ReceiveNetworkData(con,@buffer$,Len(buffer$))
          buffer$=Left(buffer$,bytes)
          If bytes<>-1 : response$=response$+buffer$ : EndIf
          ;check for end of response
          If (bytes<14500 And bytes<>-1)
            If CountString(response$,"Transfer-Encoding: chunked")=0
              If CountString(response$,"Content-Length")>0 And Len(Mid(response$,FindString(response$,#CRLF$+#CRLF$,1)+4))=Val(GetHeader(response$,"Content-Length"))
                Break
              EndIf
            Else
              ;chunk handling
              timer=ElapsedMilliseconds()
            EndIf
          EndIf
      EndSelect
      ;this will handle both chunked and timeouts
      If ElapsedMilliseconds()-timer>=timeout : CloseNetworkConnection(con) : Break : EndIf
    ForEver
    CloseNetworkConnection(con)
  EndIf
  ProcedureReturn response$
EndProcedure

Procedure.s HTTPGet(server$,port.l,headers$,timeout.l=6000)
  ;headers$ should already have #CRLF$ after each
  ;chunked responses are transparent, PB doesn't shows 0-length packet
  Protected.l con
  Protected.l timer
  Protected.l bytes
  Protected.l ret
  Protected head$
  Protected part$
  Protected buffer$
  Protected response$
  con=OpenNetworkConnection(GetURLPart(server$,#PB_URL_Site),port,#PB_Network_TCP,timeout)
  If con
    ;build header
    head$="GET /"+GetURLPart(server$,#PB_URL_Path)
    If Len(GetURLPart(server$,#PB_URL_Parameters))>0 : head$=head$+"?"+GetURLPart(server$,#PB_URL_Parameters) : EndIf
    head$=head$+" HTTP/1.1"+#CRLF$
    If Len(headers$)>0 : head$=head$+headers$ : EndIf
    head$=head$+"Host: "+GetURLPart(server$,#PB_URL_Site)+#CRLF$
    head$=head$+"Accept: */*"+#CRLF$+#CRLF$
    timer=ElapsedMilliseconds()
    Repeat
      If ret>0
        ;send remaining
        part$=Mid(head$,ret,Len(head$))
        bytes=SendNetworkData(con,@part$,Len(part$))
      Else
        ;start sending
        bytes=SendNetworkData(con,@head$,Len(head$))
      EndIf
      If bytes<>-1 : ret=ret+bytes : EndIf
      If ElapsedMilliseconds()-timer>=timeout : CloseNetworkConnection(con) : ProcedureReturn "" : EndIf
    Until ret=Len(head$)
    ret=0
    ;receive response
    timer=ElapsedMilliseconds()
    Repeat
      Delay(100)
      Select NetworkClientEvent(con)
        Case #PB_NetworkEvent_Data
          buffer$=Space(14500)
          bytes=ReceiveNetworkData(con,@buffer$,Len(buffer$))
          buffer$=Left(buffer$,bytes)
          If bytes<>-1 : response$=response$+buffer$ : EndIf
          ;check for end of response
          If (bytes<14500 And bytes<>-1)
            If CountString(response$,"Transfer-Encoding: chunked")=0
              If CountString(response$,"Content-Length")>0 And Len(Mid(response$,FindString(response$,#CRLF$+#CRLF$,1)+4))=Val(GetHeader(response$,"Content-Length"))
                Break
              EndIf
            Else
              ;chunk handling
              timer=ElapsedMilliseconds()
            EndIf
          EndIf
      EndSelect
      ;this will handle both chunked and timeouts
      If ElapsedMilliseconds()-timer>=timeout : Break : EndIf
    ForEver
    CloseNetworkConnection(con)
  EndIf
  ProcedureReturn response$
EndProcedure

SetClipboardText(HTTPGet("http://www.purebasic.com/",80,""))
;SetClipboardText(HTTPPost("http://www.purebasic.fr/english/ucp.php?mode=login",80,"username=user&password=mypass&redirect=index.php&login=Login&redirect=.%2Fucp.php%3Fmode%3Dlogin",""))
Last edited by tj1010 on Fri Apr 05, 2013 1:48 am, edited 4 times in total.
rsts
Addict
Addict
Posts: 2736
Joined: Wed Aug 24, 2005 8:39 am
Location: Southwest OH - USA

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by rsts »

This will come in handy.

Thanks for sharing.
User avatar
ostapas
Enthusiast
Enthusiast
Posts: 192
Joined: Thu Feb 18, 2010 11:10 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by ostapas »

There is something wrong with the timeout parameter in HTTPGet procedure, e.g., if it is set to 6000, then in takes all 6 seconds to download a page, if it is set to 60000, then it takes a minute and so on.
tj1010
Enthusiast
Enthusiast
Posts: 716
Joined: Mon Feb 25, 2013 5:51 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by tj1010 »

ostapas wrote:There is something wrong with the timeout parameter in HTTPGet procedure, e.g., if it is set to 6000, then in takes all 6 seconds to download a page, if it is set to 60000, then it takes a minute and so on.

It's not a bug. I just don't use Content-Length to return early. For chunked responses I'd have to use some sort of timer+buffer-check in #PB_NetworkEvent_None, because PB network functions will not give the 0-length terminator used in chunked encoding. When I figure out an intelligent way to check for the end of chunked encoding, I'll implement both and update the post.

SIDE NOTE: I also experimented with PB net functions for built in transactions and found design flaws. Like when one client has multiple streams, there is no way to track streams...
User avatar
ostapas
Enthusiast
Enthusiast
Posts: 192
Joined: Thu Feb 18, 2010 11:10 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by ostapas »

It's not a bug. I just don't use Content-Length to return early.
Aha, I suspected it is an experimental version of these procedures.
tj1010 wrote:..., I'll implement both and update the post.
Impatiently waiting :)
tj1010
Enthusiast
Enthusiast
Posts: 716
Joined: Mon Feb 25, 2013 5:51 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by tj1010 »

ostapas wrote:
It's not a bug. I just don't use Content-Length to return early.
Aha, I suspected it is an experimental version of these procedures.
tj1010 wrote:..., I'll implement both and update the post.
Impatiently waiting :)

UPDATE:
-Implemented Content-Length in non-chunked
-Made Content-Type dynamic in POST so users can set it in headers$ parameter, good if you want to do file uploads or use non-form POST APIs

EDIT: Found one example of chunked handling, in German forum. It uses a buggy timer and buffer-check method. If my solution is "experimental" I'm interested in seeing what is considered production. I see nothing usable outside of this...
User avatar
ostapas
Enthusiast
Enthusiast
Posts: 192
Joined: Thu Feb 18, 2010 11:10 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by ostapas »

Getting "Can't compare strings with numerical values" error on line 80 and 143, apparently a bracket is missing, should be

Code: Select all

 If CountString(response$,"Content-Length")>0 And Len(Mid(response$,FindString(response$,#CRLF$+#CRLF$,1)+4))=Val(GetHeader(response$,"Content-Length"))
Otherwise, thanks a lot for the code!
tj1010
Enthusiast
Enthusiast
Posts: 716
Joined: Mon Feb 25, 2013 5:51 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by tj1010 »

ostapas wrote:Getting "Can't compare strings with numerical values" error on line 80 and 143, apparently a bracket is missing, should be

Code: Select all

 If CountString(response$,"Content-Length")>0 And Len(Mid(response$,FindString(response$,#CRLF$+#CRLF$,1)+4))=Val(GetHeader(response$,"Content-Length"))
Otherwise, thanks a lot for the code!

I've updated it.

EDIT: I added code to clean up spaces by using Left() on each returned buffer with ReceiveNetworkData() returned bytes.
User avatar
RichAlgeni
Addict
Addict
Posts: 935
Joined: Wed Sep 22, 2010 1:50 am
Location: Bradenton, FL

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by RichAlgeni »

I'm not exactly sure what problem you were trying to solve? The code would run, but never return anything.
tj1010
Enthusiast
Enthusiast
Posts: 716
Joined: Mon Feb 25, 2013 5:51 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by tj1010 »

RichAlgeni wrote:I'm not exactly sure what problem you were trying to solve? The code would run, but never return anything.

I was doing a cross-platform raw HTTP solution with no API. Both work everywhere except with SSL and compression. The example copies it to clipboard.

If you give a specific URL I can find any bugs.
User avatar
RichAlgeni
Addict
Addict
Posts: 935
Joined: Wed Sep 22, 2010 1:50 am
Location: Bradenton, FL

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by RichAlgeni »

I haven't had a chance to go through it yet, but it looks like you are off to a good start. Keep striving to improve it. Change the length from hardcoded 14500, to a variable. Set the timeout much lower, then make multiple calls to receive data. Try breaking out your send into a separate procedure. That way you can reuse that procedure. It's already generic, as in not specific for web pages, so in that regard you've done a nice job!

Change you '.l' types to '.i', that way you can go to 64 bit code with no issues. It's great that you are working on cross platform code, but at some point, you may need to utilize a dll, or API. Many dlls and API calls don't pass or return strings. Try passing pointers instead. Return an integer, positive for success, negative for an error.

Keep reading, studying the forums, keep trying out new code. Never stop learning!
DarkPlayer
Enthusiast
Enthusiast
Posts: 107
Joined: Thu May 06, 2010 11:36 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by DarkPlayer »

Hello tj1010,

i just took a closer look at your code and found some flaws which should not be inside a code offering proper HTTP handling.

First of all, you should check for the #PB_NetworkEvent_Disconnect event and disable keep alive. This allows you to detect a closed connection without waiting for the timeout and you can also handle HTTP 1.0 which closes the connection after sending all data. Using keep alive while closing the socket after each transfer does not make any sense either. You also do not proper handle chunked encoded data, the size of each chunked block is still inside the output. It is not possible to distinct between real data and the chunked size field which results in an unusable output. You can use this site http://jigsaw.w3.org/HTTP/ChunkedScript to test ist. The output of your code will contain new lines and numbers which are not part of the real data. Another problem is that you are using strings and string commands to handle buffers, this makes your code unusable when enabling unicode (for example you are trying to send an unicode header to the server, or to be precise, the half of it).

There are also some minor flaws. According to rfc2616 (http://www.w3.org/Protocols/rfc2616/rfc ... l#sec14.23) the host header field must include the port number if it is not 80. You also try to determine things like the content length by searching for "Content-Length", but according to section 4.2 of rfc2612 the names of the headere are case-insensitive, so your code may not work with some http servers.

There may also be some other problems in the source code as i just took a quick view on it. You should fix this issues before advertising proper http handling.

DarkPlayer
tj1010
Enthusiast
Enthusiast
Posts: 716
Joined: Mon Feb 25, 2013 5:51 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by tj1010 »

DarkPlayer wrote:Hello tj1010,

i just took a closer look at your code and found some flaws which should not be inside a code offering proper HTTP handling.

First of all, you should check for the #PB_NetworkEvent_Disconnect event and disable keep alive. This allows you to detect a closed connection without waiting for the timeout and you can also handle HTTP 1.0 which closes the connection after sending all data. Using keep alive while closing the socket after each transfer does not make any sense either. You also do not proper handle chunked encoded data, the size of each chunked block is still inside the output. It is not possible to distinct between real data and the chunked size field which results in an unusable output. You can use this site http://jigsaw.w3.org/HTTP/ChunkedScript to test ist. The output of your code will contain new lines and numbers which are not part of the real data. Another problem is that you are using strings and string commands to handle buffers, this makes your code unusable when enabling unicode (for example you are trying to send an unicode header to the server, or to be precise, the half of it).

There are also some minor flaws. According to rfc2616 (http://www.w3.org/Protocols/rfc2616/rfc ... l#sec14.23) the host header field must include the port number if it is not 80. You also try to determine things like the content length by searching for "Content-Length", but according to section 4.2 of rfc2612 the names of the headere are case-insensitive, so your code may not work with some http servers.

There may also be some other problems in the source code as i just took a quick view on it. You should fix this issues before advertising proper http handling.

DarkPlayer

I wrote it in a few hours, sorry if it doesn't meet up to the standards of all your contributions.. oh wait

You are partially right about some things, I'll try to respond in order:
-#PB_NetworkEvent_Disconnect makes sense but only returns from failures sooner in most cases...
-HTTP 1.0 closes all types of connection after sending data? Also why implement a legacy version of anything??
-The size of each chunk isn't actually, only in TCP frame data which you can't access without API. Completed is indicated by 0 bytes as is described in RFC and everywhere else... You can't even detect 0-byte end without API...
-I admit I don't handle Unicode, I didn't realize I was trying to hide that in 160 lines of Pure Basic..
-You are right about port
-You are right about Content-Length

I will implement #PB_NetworkEvent_Disconnect, Unicode, Host: port, and Content-Length universal case handling when I get another break tonight or tomorrow.

I've searched these forums and PB archives, if you know of a more complete and/or stable solution please DO share...
DarkPlayer
Enthusiast
Enthusiast
Posts: 107
Joined: Thu May 06, 2010 11:36 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by DarkPlayer »

tj1010 wrote: I wrote it in a few hours, sorry if it doesn't meet up to the standards of all your contributions.. oh wait
[...]
I've searched these forums and PB archives, if you know of a more complete and/or stable solution please DO share...
I have written one today, you can find it here: http://www.purebasic.fr/english/viewtop ... 12&t=54297

DarkPlayer
tj1010
Enthusiast
Enthusiast
Posts: 716
Joined: Mon Feb 25, 2013 5:51 pm

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Post by tj1010 »

I recommend everyone go use that lib, there is API SSL(not sure if cross platform) and compression, and I'm smart enough to not compete over raw HTTP handling, especially when it means someone else has to support it and I can go work on other things XD
Post Reply