HTTP Library (Crossplatform, GET/HEAD/POST, chunked, gzip)

[DarkPlayer]

Hi,

I was looking for a small HTTP library fully written in PureBasic which is able to handle features like chunked data transfer encoding or gzip compression, but unfortunately most other user-written implementations only implement a very minimal set of features (or are simply broken) - so I decided to quickly write my own implementation in a few hours. I also found another bug in the network implementation, a workaround for this issue can be found below.

Supported features:
- HTTP 1.1/1.0
- GET/POST/HEAD requests
- Ascii/Unicode - 32/64 bit compatible!
- Linux / Windows support, Mac OS should work but I didn't test it.
- gzip/deflate support without any additional external libraries (uses zlib implementation bundled with PB)!
- POST: Both application/x-www-form-urlencoded (default) and multipart/form-data supported
- "chunked" Transfer-Encoding supported (used by Apache+PHP, so this is very important if you want to query your own PHP scripts)
- HTTP Basic Authentication supported (must be included in the url http://username:passwort@www...)
- Automatically follows redirects (both permanent and temporary redirects)
- Read headers or send your own raw post data!
- Read/EOF functions similar to files which allows on-the-fly reading of HTTP streams (You could feed this stream to an audio/movie decoder for example)
- For lazy users: SimpleHTTP functions, see examples at the end

Limitations:
- The POST-data is put together in one big memory block - not suitable for large files!
- When using multipart/form-data there is a very small chance that the boundary-string also appears in the transmitted data, which might confuse the server
- No https support - this library will fallback to http if you try to request https urls. YOU ARE RESPONSIBLE FOR THE SECURITY OF YOUR USERS! IF AVOIDABLE NEVER SEND PLAINTEXT PASSWORDS!

Due to bugs in Purebasics network implementation you need to use the following workaround code: http://www.purebasic.fr/english/viewtop ... 12&t=54302. This code is also included in the download package below.

Download link: https://dl.dropboxusercontent.com/u/614 ... -1.0.2.zip (md5: c549d2c6cd35a68c605826548feee510)
The included image is from the Tango base icon theme which is public domain, so you can redistribute it in your own program without any restrictions.

Changelog:
2013-04-12: Added DownloadDialog module, several enhancements in http.pbi.
2013-04-13: Improved DownloadDialog on Windows
2013-06-15: Fixed a small bug in redirection handling
2013-06-22: Fixed a small bug in URL Encoding

Example usage (see examples.pb in archive):

Code: Select all

XIncludeFile "disconnect.pbi" ;Must be in front of all HTTP Query Includes!!
XIncludeFile "http.pbi"
XIncludeFile "downloaddialog.pbi" ;Must be in front of all code using Windows!!

; ---- EXAMPLES ----

; You still have to manually initialize the network first
InitNetwork()

;- SimpleHTTP_GET
Debug "SimpleHTTP_GET Example:"

; Download a file directly to a string
Debug SimpleHTTP_GET("http://www.cs.tut.fi/cgi-bin/run/~jkorpela/echo.cgi")
Debug SimpleHTTP_GET("http://www.fds-team.de/")

;- SimpleHTTP_POST
Debug "-------------------"
Debug "SimpleHTTP_POST Example:"

NewList PostData.HTTPQuery_KeyValue()

AddElement(PostData())
PostData()\key = "key1"
PostData()\value = "value!with%special$characters&"

AddElement(PostData())
PostData()\key = "key2"
PostData()\value = "Here" + #CRLF$ + "some" + #CRLF$ + "value" + #CRLF$ + "with" + #CRLF$ + "newlines"

; Perform a POST request using the key-value pairs given above
; In this case we use multipart as encoding, but the other method also works for most scripts
Debug SimpleHTTP_POST("http://www.cs.tut.fi/cgi-bin/run/~jkorpela/echo.cgi", PostData(), #True)

;- SimpleHTTP_GET_File
Debug "-------------------"
Debug "SimpleHTTP_GET_File Example:"

; This downloads a file and on-the-fly calculates the md5 checksum.
Define filename.s = GetTemporaryDirectory() + "5MB.qsc"

Define md5.s = SimpleHTTP_GET_File("http://speedtest.qsc.de/5MB.qsc", filename)
If LCase(md5) = "8075a5bbe46e4d00d8e5eec439cf9a39"
  Debug "Checksum okay!"
Else
  Debug "Something was wrong with the download! MD5 checksum doesnt match!"
  Debug "MD5 = " + md5
EndIf

;You may want to delete this file ;-)
;DeleteFile(filename)

;- SimpleHTTP_GET_Lines
Debug "-------------------"
Debug "SimpleHTTP_GET_Lines Example:"

NewList Lines.s()

; This will perform a GET request and add each line to the output list
If SimpleHTTP_GET_Lines("http://www.cs.tut.fi/cgi-bin/run/~jkorpela/echo.cgi", Lines())
  Debug "Downloaded " + Str(ListSize(Lines())) + " lines"
  
  ; Show the lines
  ForEach Lines()
    Debug "Line: " + Lines()
  Next
  
EndIf

;- DownloadDialog
Debug "-------------------"
Debug "DownloadDialog Example:"

Define *dl = DownloadDialog_Create()

If *dl
	DownloadDialog_AddFile(*dl, "http://speedtest.qsc.de/5MB.qsc", GetTemporaryDirectory() + "5MB.qsc")
	DownloadDialog_AddFile(*dl, "http://speedtest.tweak.nl/10mb.bin", GetTemporaryDirectory() + "10mb.bin")
	;DownloadDialog_AddFile(*dl, "http://speedtest.qsc.de/10GB.qsc", "/dev/null")
	DownloadDialog_Start(*dl)
	
	While #True
		Define Event.i = WaitWindowEvent()
		; You can handle your own events here as usual ...
		
		If DownloadDialog_Status(*dl) <> #DOWNLOAD_STATUS_INPROGRESS
			Break
		EndIf
	Wend
	
	Debug "Overall download status is: " + Str(DownloadDialog_Status(*dl))
	
	While #True
		Define filename.s = DownloadDialog_NextFailedFile(*dl)
		If filename = ""
			Break
		EndIf
		
		Debug "File " + filename + " could not be downloaded"
	Wend
	
	DownloadDialog_Free(*dl)
EndIf

DarkPlayer

[DarkDragon]

Nice solution!

[tj1010]

Indeed this is a nice solution, although I've only did one test. I should go find a webkit framework and write a "proper" lib on a break too

Also I hope for the sake of your users nothing ever goes wrong with StringByteLength format handling

[True29]

Nice Lib.

[Deluxe0321]

Nice Code!
Now we need only TLS/SSL and everything is setup for a pretty straight forward http(s) client.
I think that could be achieved with OpenSSL http://savetheions.com/2010/01/16/quick ... nssl-in-c/

[DarkPlayer]

I added a non blocking download dialog in the package. You can now create a download dialog which shows the user the current progress of the download. You can also add multiple files and check the status of each file.

The window is not modal or blocking, you can continue to use your current code to handle window events without a change as the code will hook the WindowEvent or WaitWindowEvent functions to handle network data. This means you do not even need to call a special function or care about the status updates. You only need to check the status of the download to notice when it's done by calling DownloadDialog_Status().

I made a screenshot showing the download dialog running on ubuntu:



By the way: As the source is getting bigger and bigger, I decided to put all the library code in a zip archive. Take a look at the first post for a download URL.

DarkPlayer

[zxtunes.com]

Thank you very much DarkPlayer!!

But i have some bugs:

RAW POST work fine, multipart post not work.

Sample:

SimpleHTTP_POST_RAW("http://www.avito.ru/search", "location_id=652470", 1024)

NewList PostData.HTTPQuery_KeyValue()
AddElement(PostData())
PostData()\key = "location_id"
PostData()\value = "652550"
SimpleHTTP_POST("http://www.avito.ru/search", PostData(), #True)

And redirect not work.

I send post request and see:

Code: Select all

Redirecting to location '/ryazanskaya_oblast_shilovo'
<!DOCTYPE html> <head> <meta charset="utf-8">.........

Lib not redirected to adress in location (/ryazanskaya_oblast_shilovo).

[DarkPlayer]

Hi,

thanks for the feedback. You indeed discovered a small bug. The problem was not related to multipart post, but instead to the handling of POST-redirect requests. The specification says that the destination of the redirect should be queried using a GET request for statuscodes 301 and 302, this was still missing and the library also sent a POST request to the specified redirection URL. This behaviour is only intended for a 307 status code redirection and for all other redirections the GET method must be used. I have changed a few lines and it should work now.

Just try yourself and download the updated version 1.0.1 above.

DarkPlayer

[zxtunes.com]

Thx!!

I found next bug

Compare file size:

Code: Select all

a$ = SimpleHTTP_GET("http://www.avito.ru/novosibirsk", 1024*2048)
CreateFile(0, "test dark")
WriteString(0, a$)

ReceiveHTTPFile("http://www.avito.ru/novosibirsk", "test pb")

You lib download not all file.

[DarkPlayer]

Hi,

your specified website's content size differs on every request. If I execute "wget -O /dev/null http://www.avito.ru/novosibirsk" I always get a different size shown. Do you experience the same problem with a static page? If yes, you should check if you have the PB-DisconnectEvent workaround included (disconnect.pbi).

Moreover the different file size is caused by the encoding differences. ReceiveHTTPFile(...) doesn't care about encoding and just saves everything to the disk, the size you get is the number of received bytes.

The SimpleHTTP_GET(...) command internally decodes the UTF8 data to an PB String. If you are in ASCII mode, you will loose information as not all UTF-8 characters can be represented in ASCII. This only works correct if you are using Unicode support in PB and size of the string is the number of characters in the document. To get approximately the same result as with ReceiveHTTPFile(...) when using SimpleHTTP_GET(), you have to use WriteString(...) with #PB_UTF8 to reencode it (in Unicode mode).

Does this solve your issue? If not, please test SimpleHTTP_GET_File() and check if this also returns a wrong file size. I have tested it using your URLs specified and for me everything looks okay, as the string ends with the same characters as when using "wget". If you do not want the library to decode the Information, you need to use HTTPQuery_* directrly or modify SimpleHTTP_GET. Just modify the following line:

Code: Select all

result  + PeekS(*buffer, temp, #PB_UTF8)

DarkPlayer

[jamirokwai]

Hi there,

haven't tested this in real-life yet, but it works on my Mac without changes.

Great, thanks for sharing!

[zxtunes.com]

Does this solve your issue?

Yes, thx.

But i find next thing like bug.

Code: Select all

encodeURL_UTF8("АБВГД")

Return %FF%FF%FF%FF

Conversion not work for simbol >128. Test in ASCII & Unicode mode.

[DarkPlayer]

Hi,

sorry that I did not see your bug report earlier, somehow the board did not send me an email notification.

Nevertheless you are right, the RSet() command from PB, which I used in encodeURL_UTF8() did not work in all cases as expected and caused this bug. I replaced it with a different command and now characters > 128 work without any problems (see updated version above).

DarkPlayer

[Oxyandy]

Hi Guys,
Itty bit of help please
In the examples.pb
I see this example for POST but I'm a bit lost, sorry

Code: Select all

;- SimpleHTTP_POST
Debug "-------------------"
Debug "SimpleHTTP_POST Example:"

NewList PostData.HTTPQuery_KeyValue()

AddElement(PostData())
PostData()\key = "key1"
<snip>

Now I have commented out the other examples in examples.pb for the moment..

I want to add a single example to help me understand usage for UPLOAD

So, could someone show me an example for using POST to upload a FILE please

Lets call the file Example.zip which is in same folder as the (dot)PB aka current folder

Thanks very much

[DarkPlayer]

Hello,

the SimpleHTTP_Post() functions are not suitable for file upload.

There are several reasons for this: Uploading a file does often take some time and the SimpleHTTP would cause your program to freeze until all post data is sent. The better approach would be to show some progress to the user, which would involve some kind of event loop or callback. The other problem is that files are often very big and you do not want to load all the data into the memory, so you also need to handle the reading of the file content in the event/callback loop. As you can see the best solution for file upload depends on the type of application you want to write, so we decided to limit the Post Data to strings as they are small.

The SimpleHTTP functions were intended to give a fast access to the HTTP lib, but as they are always blocking, I would also recommend to directly use the internal functions if you can't guarantee that the data transfer will be very fast. You would still face the problem that also these function block till the request is sent to the server (they are non blocking while receiving the website content). They were simply not made for big data uploads, just big downloads. To get some proper support for file uploads, you would need to change this behaviour to non blocking and implement a function to send binary data.

The HTTP libary is not 100% perfect for all use cases, for example your application would freeze if the connection to the webserver is very slow and your request would take ages to be sent. We did not fix this issue as the PureBasic Network functions like OpenNetworkConnection() are also blocking and your program would most probably freeze anyway. First we would need a better PB Network Lib before it would make sense to optimize it even further in this way.

If your application only requires sending of ascii/utf8 files, and they are very small, then you can still use this library, but you have to put together the POST data by yourself. Take a look at 6. Examples in http://tools.ietf.org/html/rfc1867 to see how the data has to be constructed before sending to the server. In all other cases it is probably better to choose another library instead of this one, as it would require a lot of rewrites to solve all this problems.

DarkPlayer