Simple HTTP GET function

[mp303]

Based on an idea by Thalius, here's a simplified HTTP GET function:

Code: Select all

; HTTP_GET.pbi
; by Rasmus Schultz
; Version 1.0c

; Based on a technique by Marius Eckardt (Thalius)

#HTTP_BUFFER_SIZE = 4096  ; receive buffer size
#HTTP_IDLE_DELAY = 20     ; delay during inactivity
#HTTP_LAG_DELAY = 50      ; delay during little activity
#HTTP_TIMEOUT = 10000     ; timeout in milliseconds

#HTTP_OK = 0

#HTTP_ERROR_CONNECT   = -1     ; Unable to connect to the specified server
#HTTP_ERROR_MEMORY    = -2     ; Unable to allocate memory for response buffer
#HTTP_ERROR_TIMEOUT   = -3     ; #HTTP_TIMEOUT exceeded
#HTTP_ERROR_FILE      = -4     ; Local file could not be created
#HTTP_ERROR_PROTOCOL  = -5     ; Unknown HTTP protocol (version 1.0 or 1.1 required)

Enumeration ; Parser states
  #HTTP_STATE_EXPECT_HEADER
  #HTTP_STATE_HEADER
  #HTTP_STATE_DATA
EndEnumeration

Macro HTTP_Debug(Str)
  Debug Str ; comment out this line to disable debugging messages
EndMacro

Procedure HTTP_GET(URL.s, LocalFilename.s)
  
  ; function returns 0 on successful download (HTTP status 200)
  ; negative number in case of tehnical errors (see #HTTP_ERROR codes above)
  ; or positive number in case of HTTP status code other than "200"
  
  Protected Host.s      ; Server's hostname
  Protected Path.s      ; Remote path
  Protected Port.l = 80 ; Port number
  
  Protected Pos.l       ; Used for various string operations
  
  Protected Con.l       ; Connection ID
  
  Protected Request.s   ; HTTP request headers
  
  Protected CRLF.s = Chr(13) + Chr(10)
  
  ; Parse URL:
  
  If FindString(URL, "http://", 1) = 1 : URL = Right(URL, Len(URL)-7) : EndIf
  
  Pos = FindString(URL, "/", 1)
  If Pos = 0
    Host = URL
    Path = "/"
  Else
    Host = Left(URL, Pos-1)
    Path = Right(URL, Len(URL)-Pos+1)
  EndIf
  
  Pos = FindString(Host, ":", 1)
  If Pos > 0
    Port = Val(Right(Host, Len(Host)-Pos))
    Host = Left(Host, Pos-1)
  EndIf
  
  HTTP_Debug("Host: " + Chr(34) + Host + Chr(34))
  HTTP_Debug("Path: " + Chr(34) + Path + Chr(34))
  HTTP_Debug("Port: " + Str(Port))

  ; Allocate response buffer:
  
  Protected *Buffer
  *Buffer = AllocateMemory(#HTTP_BUFFER_SIZE)
  If Not *Buffer : ProcedureReturn #HTTP_ERROR_MEMORY : EndIf
  
  ; Open connection:
  
  Con = OpenNetworkConnection(Host, Port)
  If Con = 0 : ProcedureReturn #HTTP_ERROR_CONNECT : EndIf
  
  ; Send HTTP request:
  
  Request = "GET " + Path + " HTTP/1.0" + CRLF
  Request + "Host: " + Host + CRLF
  Request + "Connection: Close" + CRLF + CRLF
  
  SendNetworkString(Con, Request)
  
  ; Create output file:
  
  Protected FileID.l = 0
  
  ; Process response:
  
  Protected Exit.l = #False   ; Exit flag
  
  Protected Bytes.l           ; Number of bytes received
  
  Protected Time.l = ElapsedMilliseconds() ; Time of last data reception
  
  Protected Status.l = #HTTP_OK ; Status flag
  
  Protected State.l = #HTTP_STATE_EXPECT_HEADER
  
  Protected String.s          ; Parser input
  Protected Index.l           ; Parser position
  Protected Char.s            ; Parser char
  
  Protected Header.s          ; Current header
  
  Protected HTTP_Protocol.s   ; HTTP protocol version
  Protected HTTP_Status.l = 0 ; HTTP status code
  
  Protected Redirected.b = #False
  
  Repeat
    
    If NetworkClientEvent(Con) = #PB_NetworkEvent_Data
      
      Repeat
        
        Bytes = ReceiveNetworkData(Con, *Buffer, #HTTP_BUFFER_SIZE)
        
        If Bytes = 0
          
          Exit = #True
          
        Else
          
          If Bytes < #HTTP_BUFFER_SIZE : Delay(#HTTP_LAG_DELAY) : EndIf
          
          HTTP_Debug("Received: " + Str(Bytes) + " bytes")
          
          If State = #HTTP_STATE_DATA
            
            WriteData(FileID, *Buffer, Bytes)
            
          Else
            
            String = PeekS(*Buffer, Bytes, #PB_Ascii)
            Index = 0
            
            Repeat
              
              Index + 1
              Char = Mid(String, Index, 1)
              
              Select State
                
                Case #HTTP_STATE_EXPECT_HEADER
                  If Char = Chr(10)
                    State = #HTTP_STATE_DATA
                    HTTP_Debug("Creating file: " + LocalFilename)
                    FileID = CreateFile(#PB_Any, LocalFilename)
                    If FileID = 0
                      Exit = #True
                      Status = #HTTP_ERROR_FILE
                    ElseIf Index < Bytes
                      WriteData(FileID, *Buffer+Index, Bytes-Index)
                    EndIf
                  ElseIf Char = Chr(13)
                    ; (ignore)
                  Else
                    Header = Char
                    State = #HTTP_STATE_HEADER
                  EndIf
                
                Case #HTTP_STATE_HEADER
                  If Char = Chr(10)
                    If HTTP_Status = 0
                      HTTP_Protocol = StringField(StringField(Header, 1, " "), 2, "/")
                      HTTP_Status = Val(StringField(Header, 2, " "))
                      If ((HTTP_Protocol <> "1.0") And (HTTP_Protocol <> "1.1")) Or (StringField(StringField(Header, 1, " "), 1, "/") <> "HTTP") Or (HTTP_Status = 0)
                        HTTP_Debug("HTTP Protocol error!")
                        Exit = #True
                        Status = #HTTP_ERROR_PROTOCOL
                      EndIf
                      HTTP_Debug("HTTP Protocol " + HTTP_Protocol + ", Status " + Str(HTTP_Status))
                      If (HTTP_Status >= 300) And (HTTP_Status < 400)
                        HTTP_Debug("Redirection...")
                        Redirected = #True
                      ElseIf HTTP_Status <> 200
                        Status = HTTP_Status
                        Exit = #True
                        HTTP_Debug("Status <> 200 - abort!")
                      EndIf
                    ElseIf Left(Header, 10) = "Location: "
                      Status = HTTP_GET(Right(Header, Len(Header)-10), LocalFilename)
                      Exit = #True
                    Else
                      HTTP_Debug(Header)
                    EndIf
                    State = #HTTP_STATE_EXPECT_HEADER
                  ElseIf Char = Chr(13)
                    ; (ignore)
                  Else
                    Header + Char
                  EndIf
                
              EndSelect
              
            Until (State = #HTTP_STATE_DATA) Or (Index = Bytes) Or (Exit = #True)
            
          EndIf
          
          Time = ElapsedMilliseconds()
          
        EndIf
        
      Until Exit = #True
      
    Else
      
      HTTP_Debug("Idle...")
      Delay(#HTTP_IDLE_DELAY)
      
      If ElapsedMilliseconds() - Time > #HTTP_TIMEOUT
        Exit = #True
        Status = #HTTP_ERROR_TIMEOUT
      EndIf
      
    EndIf
    
  Until Exit = #True
  
  ; Close and finish:
  
  CloseNetworkConnection(Con)
  FreeMemory(*Buffer)
  If FileID <> 0 : CloseFile(FileID) : EndIf
  
  ProcedureReturn Status
  
EndProcedure

And here's an example:

Code: Select all

; --- Example:

If InitNetwork() = 0
  Debug "No TCP provider..."
Else
  Status.l = HTTP_GET("http://www.wavemage.com/edscore/5-EndTitle.mp3", GetCurrentDirectory()+"test.mp3")
  Debug "DownloadFile returned: " + Str(Status)
EndIf

The function returns zero on success - negative return values indicate a technical error, as indicated by the #HTTP_ERROR-constants. Positive return values indicate a HTTP status code, e.g. the server reported an error.

The function will follow redirects.

Edit: minor fixes
Edit: local variables declared with Protected instead of Define

[Thalius]

nice !!

Got to rework mine for HTTP 1.0 probably adding some stuff ( redirects also ) hehe. Nice code btw ( much more readable than mine - if i fiddle such stuff togetehr i usually do it on a widescreen hence teh long lines at times =P ).

Lil thing just about general : Define is usually just used outside procedures to define a for example global varspace - or set the type. In a procedure it makes more sense to define thoose vars inside a Protected keyword - that way it shouldnt collide within a thread or a global keyword.

Nice Work,
Thalius

[mp303]

Lil thing just about general : Define is usually just used outside procedures to define a for example global varspace - or set the type. In a procedure it makes more sense to define thoose vars inside a Protected keyword - that way it shouldnt collide within a thread or a global keyword. :)

Nice Work,
Thalius

Code updated (above) using Protected instead of Define.

I thought the point of using Define was that variables had local scope? Guess not. The help file is not very clear on this - not sure I understand the difference between Global and Define, they both declare global variables, it seems..??

[Thalius]

no, not global. Define alone defines local vars.

Usually used to Define types and vars when enableexplicit is used.

Kind of like :

a.f

Its not wrong in this case but hurts the eyes seeing define in a procedure somehow. protecting cant hurt either especially if somone plans to use threads.

Thalius

[mp303]

no, not global. Define alone defines local vars.

so what's wrong with using define then? even if the application using this function is multi-threaded, I don't see why regular local variables should cause problems? the function itself is not multi-threaded.

[Thalius]

just a matter of style basically ( take a look at C ) =)
as i said above: Its correctly working but still cant hurt to get a habit of certain standards =)

Thalius