Page 1 of 2

Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Thu Apr 04, 2013 9:54 am
by tj1010
I really needed stable cross-platform HTTP procedures I could just drop in source code, and could only find short unstable examples here and in archives.. I wrote this today and tested it with chunking&non-chunking HTTP POST&GET servers. Call them with Location header value for 301 and 302 responses.

If you want SSL and/or compression I suggest API, although you have to use bloated COM for flexible POST on Windows.

Let me know if you hit issues

Code: Select all

;Improvements to be made:
;-Handle SendNetworkData limit
;-Handle 301 and 302 transparantly by parsing and calling selves
;-Better default timeout value?
EnableExplicit
InitNetwork()

Procedure.s GetHeader(Packet$,Name$)
  ProcedureReturn Mid(Packet$,FindString(Packet$,Name$)+Len(Name$)+2,FindString(Packet$,#CRLF$,FindString(Packet$,Name$))-(FindString(Packet$,Name$)+Len(Name$)+2))
EndProcedure

Procedure.s HTTPPost(server$,port.l,vars$,headers$,timeout.l=6000)
  ;headers$ should already have #CRLF$ after each
  ;variables should already be encoded in server$ and vars$
  ;chunked responses are transparent, PB doesn't show 0-length packet
  Protected con.l
  Protected head$
  Protected part$
  Protected ret.l
  Protected bytes.l
  Protected timer.l
  Protected buffer$
  Protected response$
  con=OpenNetworkConnection(GetURLPart(server$,#PB_URL_Site),port,#PB_Network_TCP,timeout)
  If con
    ;build header
    head$="POST /"+GetURLPart(server$,#PB_URL_Path)
    If Len(GetURLPart(server$,#PB_URL_Parameters))>0 : head$=head$+"?"+GetURLPart(server$,#PB_URL_Parameters) : EndIf
    head$=head$+" HTTP/1.1"+#CRLF$
    If Len(headers$)>0 : head$=head$+headers$ : EndIf
    head$=head$+"Host: "+GetURLPart(server$,#PB_URL_Site)+#CRLF$
    head$=head$+"Accept: */*"+#CRLF$
    If CountString(headers$,"Content-Type:")=0 : head$=head$+"Content-Type: application/x-www-form-urlencoded"+#CRLF$ : EndIf
    head$=head$+"Connection: keep-alive"+#CRLF$
    head$=head$+"Content-Length: "+Str(Len(vars$))+#CRLF$
    head$=head$+#CRLF$
    ;send header
    timer=ElapsedMilliseconds()
    Repeat
      If ret>0
        ;send remaining
        part$=Mid(head$,ret,Len(head$))
        bytes=SendNetworkData(con,@part$,Len(part$))
      Else
        ;start sending
        bytes=SendNetworkData(con,@head$,Len(head$))
      EndIf
      If bytes<>-1 : ret=ret+bytes : EndIf
      If ElapsedMilliseconds()-timer>=timeout : CloseNetworkConnection(con) : ProcedureReturn "" : EndIf
    Until ret=Len(head$)
    ret=0
    ;send content
    timer=ElapsedMilliseconds()
    Repeat
      If ret>0
        ;send remaining
        part$=Mid(vars$,ret,Len(vars$))
        bytes=SendNetworkData(con,@part$,Len(part$))
      Else
        ;start sending
        bytes=SendNetworkData(con,@vars$,Len(vars$))
      EndIf
      If bytes<>-1 : ret=ret+bytes : EndIf
      If ElapsedMilliseconds()-timer>=timeout : CloseNetworkConnection(con) : ProcedureReturn "" : EndIf
    Until ret=Len(vars$)
    ret=0
    ;receive response
    timer=ElapsedMilliseconds()
    Repeat
      Delay(100)
      Select NetworkClientEvent(con)
        Case #PB_NetworkEvent_Data
          buffer$=Space(14500)
          bytes=ReceiveNetworkData(con,@buffer$,Len(buffer$))
          buffer$=Left(buffer$,bytes)
          If bytes<>-1 : response$=response$+buffer$ : EndIf
          ;check for end of response
          If (bytes<14500 And bytes<>-1)
            If CountString(response$,"Transfer-Encoding: chunked")=0
              If CountString(response$,"Content-Length")>0 And Len(Mid(response$,FindString(response$,#CRLF$+#CRLF$,1)+4))=Val(GetHeader(response$,"Content-Length"))
                Break
              EndIf
            Else
              ;chunk handling
              timer=ElapsedMilliseconds()
            EndIf
          EndIf
      EndSelect
      ;this will handle both chunked and timeouts
      If ElapsedMilliseconds()-timer>=timeout : CloseNetworkConnection(con) : Break : EndIf
    ForEver
    CloseNetworkConnection(con)
  EndIf
  ProcedureReturn response$
EndProcedure

Procedure.s HTTPGet(server$,port.l,headers$,timeout.l=6000)
  ;headers$ should already have #CRLF$ after each
  ;chunked responses are transparent, PB doesn't shows 0-length packet
  Protected.l con
  Protected.l timer
  Protected.l bytes
  Protected.l ret
  Protected head$
  Protected part$
  Protected buffer$
  Protected response$
  con=OpenNetworkConnection(GetURLPart(server$,#PB_URL_Site),port,#PB_Network_TCP,timeout)
  If con
    ;build header
    head$="GET /"+GetURLPart(server$,#PB_URL_Path)
    If Len(GetURLPart(server$,#PB_URL_Parameters))>0 : head$=head$+"?"+GetURLPart(server$,#PB_URL_Parameters) : EndIf
    head$=head$+" HTTP/1.1"+#CRLF$
    If Len(headers$)>0 : head$=head$+headers$ : EndIf
    head$=head$+"Host: "+GetURLPart(server$,#PB_URL_Site)+#CRLF$
    head$=head$+"Accept: */*"+#CRLF$+#CRLF$
    timer=ElapsedMilliseconds()
    Repeat
      If ret>0
        ;send remaining
        part$=Mid(head$,ret,Len(head$))
        bytes=SendNetworkData(con,@part$,Len(part$))
      Else
        ;start sending
        bytes=SendNetworkData(con,@head$,Len(head$))
      EndIf
      If bytes<>-1 : ret=ret+bytes : EndIf
      If ElapsedMilliseconds()-timer>=timeout : CloseNetworkConnection(con) : ProcedureReturn "" : EndIf
    Until ret=Len(head$)
    ret=0
    ;receive response
    timer=ElapsedMilliseconds()
    Repeat
      Delay(100)
      Select NetworkClientEvent(con)
        Case #PB_NetworkEvent_Data
          buffer$=Space(14500)
          bytes=ReceiveNetworkData(con,@buffer$,Len(buffer$))
          buffer$=Left(buffer$,bytes)
          If bytes<>-1 : response$=response$+buffer$ : EndIf
          ;check for end of response
          If (bytes<14500 And bytes<>-1)
            If CountString(response$,"Transfer-Encoding: chunked")=0
              If CountString(response$,"Content-Length")>0 And Len(Mid(response$,FindString(response$,#CRLF$+#CRLF$,1)+4))=Val(GetHeader(response$,"Content-Length"))
                Break
              EndIf
            Else
              ;chunk handling
              timer=ElapsedMilliseconds()
            EndIf
          EndIf
      EndSelect
      ;this will handle both chunked and timeouts
      If ElapsedMilliseconds()-timer>=timeout : Break : EndIf
    ForEver
    CloseNetworkConnection(con)
  EndIf
  ProcedureReturn response$
EndProcedure

SetClipboardText(HTTPGet("http://www.purebasic.com/",80,""))
;SetClipboardText(HTTPPost("http://www.purebasic.fr/english/ucp.php?mode=login",80,"username=user&password=mypass&redirect=index.php&login=Login&redirect=.%2Fucp.php%3Fmode%3Dlogin",""))

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Thu Apr 04, 2013 1:08 pm
by rsts
This will come in handy.

Thanks for sharing.

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Thu Apr 04, 2013 2:03 pm
by ostapas
There is something wrong with the timeout parameter in HTTPGet procedure, e.g., if it is set to 6000, then in takes all 6 seconds to download a page, if it is set to 60000, then it takes a minute and so on.

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Thu Apr 04, 2013 6:49 pm
by tj1010
ostapas wrote:There is something wrong with the timeout parameter in HTTPGet procedure, e.g., if it is set to 6000, then in takes all 6 seconds to download a page, if it is set to 60000, then it takes a minute and so on.

It's not a bug. I just don't use Content-Length to return early. For chunked responses I'd have to use some sort of timer+buffer-check in #PB_NetworkEvent_None, because PB network functions will not give the 0-length terminator used in chunked encoding. When I figure out an intelligent way to check for the end of chunked encoding, I'll implement both and update the post.

SIDE NOTE: I also experimented with PB net functions for built in transactions and found design flaws. Like when one client has multiple streams, there is no way to track streams...

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Thu Apr 04, 2013 7:05 pm
by ostapas
It's not a bug. I just don't use Content-Length to return early.
Aha, I suspected it is an experimental version of these procedures.
tj1010 wrote:..., I'll implement both and update the post.
Impatiently waiting :)

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Thu Apr 04, 2013 7:57 pm
by tj1010
ostapas wrote:
It's not a bug. I just don't use Content-Length to return early.
Aha, I suspected it is an experimental version of these procedures.
tj1010 wrote:..., I'll implement both and update the post.
Impatiently waiting :)

UPDATE:
-Implemented Content-Length in non-chunked
-Made Content-Type dynamic in POST so users can set it in headers$ parameter, good if you want to do file uploads or use non-form POST APIs

EDIT: Found one example of chunked handling, in German forum. It uses a buggy timer and buffer-check method. If my solution is "experimental" I'm interested in seeing what is considered production. I see nothing usable outside of this...

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Thu Apr 04, 2013 8:26 pm
by ostapas
Getting "Can't compare strings with numerical values" error on line 80 and 143, apparently a bracket is missing, should be

Code: Select all

 If CountString(response$,"Content-Length")>0 And Len(Mid(response$,FindString(response$,#CRLF$+#CRLF$,1)+4))=Val(GetHeader(response$,"Content-Length"))
Otherwise, thanks a lot for the code!

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Thu Apr 04, 2013 8:53 pm
by tj1010
ostapas wrote:Getting "Can't compare strings with numerical values" error on line 80 and 143, apparently a bracket is missing, should be

Code: Select all

 If CountString(response$,"Content-Length")>0 And Len(Mid(response$,FindString(response$,#CRLF$+#CRLF$,1)+4))=Val(GetHeader(response$,"Content-Length"))
Otherwise, thanks a lot for the code!

I've updated it.

EDIT: I added code to clean up spaces by using Left() on each returned buffer with ReceiveNetworkData() returned bytes.

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Tue Apr 09, 2013 3:04 am
by RichAlgeni
I'm not exactly sure what problem you were trying to solve? The code would run, but never return anything.

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Tue Apr 09, 2013 5:32 am
by tj1010
RichAlgeni wrote:I'm not exactly sure what problem you were trying to solve? The code would run, but never return anything.

I was doing a cross-platform raw HTTP solution with no API. Both work everywhere except with SSL and compression. The example copies it to clipboard.

If you give a specific URL I can find any bugs.

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Tue Apr 09, 2013 7:32 pm
by RichAlgeni
I haven't had a chance to go through it yet, but it looks like you are off to a good start. Keep striving to improve it. Change the length from hardcoded 14500, to a variable. Set the timeout much lower, then make multiple calls to receive data. Try breaking out your send into a separate procedure. That way you can reuse that procedure. It's already generic, as in not specific for web pages, so in that regard you've done a nice job!

Change you '.l' types to '.i', that way you can go to 64 bit code with no issues. It's great that you are working on cross platform code, but at some point, you may need to utilize a dll, or API. Many dlls and API calls don't pass or return strings. Try passing pointers instead. Return an integer, positive for success, negative for an error.

Keep reading, studying the forums, keep trying out new code. Never stop learning!

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Wed Apr 10, 2013 2:27 am
by DarkPlayer
Hello tj1010,

i just took a closer look at your code and found some flaws which should not be inside a code offering proper HTTP handling.

First of all, you should check for the #PB_NetworkEvent_Disconnect event and disable keep alive. This allows you to detect a closed connection without waiting for the timeout and you can also handle HTTP 1.0 which closes the connection after sending all data. Using keep alive while closing the socket after each transfer does not make any sense either. You also do not proper handle chunked encoded data, the size of each chunked block is still inside the output. It is not possible to distinct between real data and the chunked size field which results in an unusable output. You can use this site http://jigsaw.w3.org/HTTP/ChunkedScript to test ist. The output of your code will contain new lines and numbers which are not part of the real data. Another problem is that you are using strings and string commands to handle buffers, this makes your code unusable when enabling unicode (for example you are trying to send an unicode header to the server, or to be precise, the half of it).

There are also some minor flaws. According to rfc2616 (http://www.w3.org/Protocols/rfc2616/rfc ... l#sec14.23) the host header field must include the port number if it is not 80. You also try to determine things like the content length by searching for "Content-Length", but according to section 4.2 of rfc2612 the names of the headere are case-insensitive, so your code may not work with some http servers.

There may also be some other problems in the source code as i just took a quick view on it. You should fix this issues before advertising proper http handling.

DarkPlayer

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Wed Apr 10, 2013 3:00 am
by tj1010
DarkPlayer wrote:Hello tj1010,

i just took a closer look at your code and found some flaws which should not be inside a code offering proper HTTP handling.

First of all, you should check for the #PB_NetworkEvent_Disconnect event and disable keep alive. This allows you to detect a closed connection without waiting for the timeout and you can also handle HTTP 1.0 which closes the connection after sending all data. Using keep alive while closing the socket after each transfer does not make any sense either. You also do not proper handle chunked encoded data, the size of each chunked block is still inside the output. It is not possible to distinct between real data and the chunked size field which results in an unusable output. You can use this site http://jigsaw.w3.org/HTTP/ChunkedScript to test ist. The output of your code will contain new lines and numbers which are not part of the real data. Another problem is that you are using strings and string commands to handle buffers, this makes your code unusable when enabling unicode (for example you are trying to send an unicode header to the server, or to be precise, the half of it).

There are also some minor flaws. According to rfc2616 (http://www.w3.org/Protocols/rfc2616/rfc ... l#sec14.23) the host header field must include the port number if it is not 80. You also try to determine things like the content length by searching for "Content-Length", but according to section 4.2 of rfc2612 the names of the headere are case-insensitive, so your code may not work with some http servers.

There may also be some other problems in the source code as i just took a quick view on it. You should fix this issues before advertising proper http handling.

DarkPlayer

I wrote it in a few hours, sorry if it doesn't meet up to the standards of all your contributions.. oh wait

You are partially right about some things, I'll try to respond in order:
-#PB_NetworkEvent_Disconnect makes sense but only returns from failures sooner in most cases...
-HTTP 1.0 closes all types of connection after sending data? Also why implement a legacy version of anything??
-The size of each chunk isn't actually, only in TCP frame data which you can't access without API. Completed is indicated by 0 bytes as is described in RFC and everywhere else... You can't even detect 0-byte end without API...
-I admit I don't handle Unicode, I didn't realize I was trying to hide that in 160 lines of Pure Basic..
-You are right about port
-You are right about Content-Length

I will implement #PB_NetworkEvent_Disconnect, Unicode, Host: port, and Content-Length universal case handling when I get another break tonight or tomorrow.

I've searched these forums and PB archives, if you know of a more complete and/or stable solution please DO share...

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Thu Apr 11, 2013 3:37 am
by DarkPlayer
tj1010 wrote: I wrote it in a few hours, sorry if it doesn't meet up to the standards of all your contributions.. oh wait
[...]
I've searched these forums and PB archives, if you know of a more complete and/or stable solution please DO share...
I have written one today, you can find it here: http://www.purebasic.fr/english/viewtop ... 12&t=54297

DarkPlayer

Re: Proper HTTP GET&POST(cross-platform,chunking etc..)

Posted: Fri Apr 12, 2013 12:31 am
by tj1010
I recommend everyone go use that lib, there is API SSL(not sure if cross platform) and compression, and I'm smart enough to not compete over raw HTTP handling, especially when it means someone else has to support it and I can go work on other things XD