Having some trouble compressing Web Pages

[RichAlgeni]

I thought that ZLib was a derivative of GZip, so I used that first. I also tried the included #PB_Packer_Zip CompressMemory command as Deflate. Both compressed the page, but neither were recognized by a browser.

Has anyone worked with this successfully?

[JHPJHP]

Found these posts - is this what your looking for?
(second post is informative - but not sure it still applies)

http://www.purebasic.fr/english/viewtopic.php?t=37984

viewtopic.php?t=36620

[RichAlgeni]

Not sure. I used the zlib_64.lib import and invaluable source from Luis. The zlib is doing a great job at compressing. I'm adding 'Content-Encoding: gzip' to the header as specified, but the compressed data sent after the header is not recognized.

[fizban]

In one of my projects, I had to do the reverse, that is, to uncompress an gzipped http content. The easiest way I found was to create a temporary file, use the gzread function and delete the temporary file. I imagine that in order to compress the stream you could use the same technique, but using gzwrite.

[RichAlgeni]

I will check that out, thanks @fizban!

[RichAlgeni]

Just for FYI, I've used the zlib.lib included with PureBasic, as well as zlib_32.lib and zlib_64.lib from the forum. None of these libraries create a compressed file that is recognizable by browsers. They all compress files into a smaller format than that of 'gzip' in a web server. To test this, I created a 60k text only web page, and loaded it on to a number of different web servers. One by Yahoo, one by Google, one by 1and1.com, as well as a couple of others. They all compressed the file down to about 3.5 KB. The instances of Zlib compress the same file down to about 3KB, and are not recognized by browsers.

[fizban]

That's weird. Note that I have a site where I pre-compress the static css and js files in order to reduce the load of the server. To do that, I simply gzip those files and modify the .htaccess to redirect the requests to the css and js files to their css.gz and js.gz counterparts. I can suggest you test your compressed file the same way. If it works, the problem is not in the compression algorithm, but in a different part of the code...
my .htaccess looks like this:

Code: Select all

AddEncoding x-gzip .gz
RewriteEngine On
RewriteCond %{REQUEST_FILENAME}.gz -f
RewriteCond %{REQUEST_FILENAME} \.css$ [OR] 
RewriteCond %{REQUEST_FILENAME} \.js$
RewriteRule (.*)$ $1.gz
<FilesMatch "\\.css.gz$">
ForceType text/css
Header set Content-Encoding: gzip
</FilesMatch>

<FilesMatch "\\.js.gz$">
ForceType text/js
Header set Content-Encoding: gzip
</FilesMatch>

[RichAlgeni]

What do you use as your web server Fizban? I have written my own server because of the circumstances. I tested the compressed file created by ZLib, and that created by mainstream web servers by creating a text only web page of 60KB, then I used a PB test 'GET' process to get the data served. I stripped off the header, and saved the remainder as a '.gz' file and was able to open it. I then took the uncompressed data, and compressed it with one of a number of ZLib libraries found in PureBasic or the forums.The ZLib compressed file was smaller than the '.gz' file, and was not recognized as a valid '.gz' file by winrar.

Do you have a GZip.exe process?

[fizban]

I tipically use Apache. The XAMPP package is good for development and testing.

For the project I mentioned, I actually used zlibstat.lib importing the required functions:

Code: Select all

Import "ZlibStat.lib" 
  gzclose(a.l) As "_gzclose@4" 
  gzopen(a.l,b.l) As "_gzopen@8" 
  gzread(a.l,b.l,c.l) As "_gzread@12" 
EndImport 

I guess for writing it would be:

Code: Select all

gzwrite(a.l,b.l,c.l) As "_gzwrite@12" 

I am not sure which zlibstat.lib I used. Most likely I compiled it myself. I uploaded it here.

[idle]

just for reference a quick hack of atomic web server demo from 4.61
save the source where ever
create a dir under it as www
save a html page in it as index.html
run it
browse to 127.0.0.1:1081

as long as the accept-encoding request field says deflate you can use it

Code: Select all

;
; ------------------------------------------------------------
;
;       Atomic Web Server in PureBasic by AlphaSND
;
;           (c) 2001 - Fantaisie Software
;
; ------------------------------------------------------------
;
; 25/03/2001
;   Added path relative feature. It can be started everywhere
;   Added the window to kill it easely
;
; 19/03/2001
;   Added some new features (clients closed automatically)
;
; 17/03/2001
;   First version.
;
CompilerIf #PB_Compiler_OS = #PB_OS_Windows 
  
  ImportC "zlib.lib"
    compress2(*dest,*destlen,*source,sourcelen,level)
    uncompress(*dest,*destlen,*source,sourcelen)
  EndImport 
CompilerElse 
  
  ImportC "-lz"   
     compress(*dest,*destlen,*source,Srclen)
     compress2(*dest,*destlen,*source,sourcelen,level)
     uncompress(*dest,*destlen,*source,sourcelen)
 EndImport 
  
CompilerEndIf 



If InitNetwork() = 0
  MessageRequester("Error", "Can't initialize the network !", 0)
  End
EndIf

server=1
Port = 8081
BaseDirectory$ = "www/"
DefaultPage$   = "Index.html"
AtomicTitle$   = "Atomic Web Server v1.0"

Global EOL$

EOL$ = Chr(13)+Chr(10)

*Buffer = AllocateMemory(10000)

If CreateNetworkServer(1,Port)

  OpenWindow(0, 0, 0, 300, 0, "Atomic Web Server (Port "+Str(Port)+")",#PB_Window_SystemMenu | #PB_Window_ScreenCentered)

  Repeat

    Repeat
      WEvent = WindowEvent()

      If WEvent = #PB_Event_CloseWindow : Quit = 1 : EndIf
    Until WEvent = 0

    SEvent = NetworkServerEvent()

    If SEvent

      ClientID.l = EventClient();NetworkClientID()

      Select SEvent

        Case 1  ; When a new client has been connected...
          Debug "New connection"

        Case 4  ; When a client has closed the connection...
          Debug "Deconnection"

        Default
          Debug "Raw data"
          RequestLength.l = ReceiveNetworkData(ClientID, *Buffer, 2000)
         ; Debug PeekS(*buffer,RequestLength) 
          
          Gosub ProcessRequest

      EndSelect

    Else
      Delay(20)  ; Don't stole the whole CPU !
    EndIf

  Until Quit = 1

  CloseNetworkServer(1)
Else
  MessageRequester(AtomicTitle$, "Error: can't create the server (port in use ?).", 0)
EndIf

End



Procedure.l BuildRequestHeader(*Buffer, DataLength.l, ContentType$)

  Length = PokeS(*Buffer, "HTTP/1.1 200 OK"+EOL$)                     : *Buffer+Length
  Length = PokeS(*Buffer, "Date: Wed, 07 Aug 1996 11:15:43 GMT"+EOL$) : *Buffer+Length
  Length = PokeS(*Buffer, "Server: Atomic Web Server 0.2b"+EOL$)      : *Buffer+Length
  Length = PokeS(*Buffer, "Content-Length: "+Str(DataLength)+EOL$)    : *Buffer+Length
  Length = PokeS(*Buffer, "Content-Type: "+ContentType$+EOL$)         : *Buffer+Length
  Length = PokeS(*Buffer, "Content-Encoding: deflate" + EOL$) : *Buffer+Length 
  Length = PokeS(*Buffer, EOL$)                                       : *Buffer+Length
 
  ProcedureReturn *Buffer
EndProcedure


ProcessRequest:

  a$ = PeekS(*Buffer)

  If Left(a$, 3) = "GET"

    MaxPosition = FindString(a$, Chr(13), 5)
    Position = FindString(a$, " ", 5)
    If Position < MaxPosition
      RequestedFile$ = Mid(a$, 6, Position-5)      ; Automatically remove the leading '/'
      RequestedFile$ = RTrim(RequestedFile$)
    Else
      RequestedFile$ = Mid(a$, 6, MaxPosition-5)   ; When a command like 'GET /' is sent..
    EndIf

      ; The following routine transforme all '/' in '\' (Windows format)
      ;
      Structure tmp
        a.b
      EndStructure

      If RequestedFile$ = ""
        RequestedFile$ = DefaultPage$
      Else
        *t.tmp = @RequestedFile$
        While *t\a <> 0
          If *t\a = '/' : *t\a = '\' : EndIf
          *t+1
        Wend
      EndIf

      ; Test if the file exists, and if not display the error message
      ;

      If ReadFile(0, BaseDirectory$+RequestedFile$)

        FileLength = Lof(0)

        Select LCase(Right(RequestedFile$,4))
          Case ".gif"
            ContentType$ = "image/gif"

          Case ".jpg"
            ContentType$ = "image/jpeg"

          Case ".txt"
            ContentType$ = "text/plain"

          Case ".zip"
            ContentType$ = "application/zip"

          Default
            ContentType$ = "text/html"

        EndSelect
     
        
        *src = AllocateMemory(FileLength) 
        ReadData(0,*src,FileLength)
        *dest = AllocateMemory(FileLength*1.5) 
        destlen = FileLength*1.5 
        compress2(*dest,@destlen,*src,FileLength,9) 
               
        *FileBuffer = AllocateMemory(destlen+500)
        *BufferOffset = BuildRequestHeader(*FileBuffer, destlen, ContentType$)
        
        Debug PeekS(*FileBuffer,*BufferOffset-*FileBuffer)
        headerlen = *BufferOffset-*FileBuffer 
        
        CopyMemory(*dest,*FileBuffer+headerlen,destlen)
        FileLength = destlen + headerlen       
        Debug "header len " + headerlen
        Debug "compressed page  " + FileLength 
                
        CloseFile(0)

        SendNetworkData(ClientID, *FileBuffer, FileLength)
        FreeMemory(*FileBuffer)
        FreeMemory(*src)
        FreeMemory(*dest)
        
      Else
        If ReadFile(0, BaseDirectory$+"AtomicWebServer_Error.html")
          FileLength = Lof(0)
          ContentType$ = "text/html"

          *FileBuffer   = AllocateMemory(FileLength+200)
          *BufferOffset = BuildRequestHeader(*FileBuffer, FileLength, ContentType$)

          ReadData(0,*BufferOffset, FileLength)
          CloseFile(0)

          SendNetworkData(ClientID, *FileBuffer, *BufferOffset-*FileBuffer+FileLength)
          FreeMemory(*FileBuffer)
        EndIf
      EndIf
  EndIf

Return

[PB]

Code: Select all

Select Right(RequestedFile$,4)

This should really be:

Code: Select all

Select LCase(Right(RequestedFile$,4))

Because not all file extensions are lowercase.

[idle]

Code: Select all

Select Right(RequestedFile$,4)

This should really be:

Code: Select all

Select LCase(Right(RequestedFile$,4))

Because not all file extensions are lowercase.

updated it thanks

[fizban]

Apparently, according to the zlib manual if you actually want to use gzip instead of deflate, you should call deflateInit2 with a specific value in the windowBits parameter

Code: Select all

#windowBits 15
#GZIP_ENCODING 16
#Z_DEFAULT_COMPRESSION  -1
#Z_DEFAULT_STRATEGY    0
#Z_DEFLATED   8
deflateInit2 (strm, #Z_DEFAULT_COMPRESSION, #Z_DEFLATED,
                             #windowBits | #GZIP_ENCODING, 8,
                             #Z_DEFAULT_STRATEGY)

and strm needs to be a z_stream. You can find a C example here

[RichAlgeni]

Perfect!

Thanks Idle!!!

I owe you one!

[RichAlgeni]

Idle is going to put a hit out on me after this, but here goes:

This whole issue came about because of data being lost from a sever to a client. Research has led me to believe this was the result of a 'router black hole.' It turned out that it was being caused by a old VPN product. We were able to bypass the problem by using 'Deflate', and code suggested by Idle.

For those not up to speed, this was a download of data into a web form, so that reports could be completed. The data is being sent out over a Motorola radio system, using a product called RadioIP.

Kludgy enough? That's what I thought! Then I asked them 'how do you get the filled out reports to the office?'

Ready for the answer?!?!

They print them out, and mail them in!!!

After all the work, research, and input from some great people, they use the mail to send in their reports!

Should have known!