Page 1 of 2

Having some trouble compressing Web Pages

Posted: Wed Aug 07, 2013 4:38 pm
by RichAlgeni
I thought that ZLib was a derivative of GZip, so I used that first. I also tried the included #PB_Packer_Zip CompressMemory command as Deflate. Both compressed the page, but neither were recognized by a browser.

Has anyone worked with this successfully?

Re: Having some trouble compressing Web Pages

Posted: Wed Aug 07, 2013 4:56 pm
by JHPJHP
Found these posts - is this what your looking for?
(second post is informative - but not sure it still applies)

http://www.purebasic.fr/english/viewtopic.php?t=37984

viewtopic.php?t=36620

Re: Having some trouble compressing Web Pages

Posted: Wed Aug 07, 2013 6:18 pm
by RichAlgeni
Not sure. I used the zlib_64.lib import and invaluable source from Luis. The zlib is doing a great job at compressing. I'm adding 'Content-Encoding: gzip' to the header as specified, but the compressed data sent after the header is not recognized.

Re: Having some trouble compressing Web Pages

Posted: Fri Aug 09, 2013 11:44 pm
by fizban
In one of my projects, I had to do the reverse, that is, to uncompress an gzipped http content. The easiest way I found was to create a temporary file, use the gzread function and delete the temporary file. I imagine that in order to compress the stream you could use the same technique, but using gzwrite.

Re: Having some trouble compressing Web Pages

Posted: Sat Aug 10, 2013 12:56 am
by RichAlgeni
I will check that out, thanks @fizban!

Re: Having some trouble compressing Web Pages

Posted: Sat Aug 10, 2013 2:21 am
by RichAlgeni
Just for FYI, I've used the zlib.lib included with PureBasic, as well as zlib_32.lib and zlib_64.lib from the forum. None of these libraries create a compressed file that is recognizable by browsers. They all compress files into a smaller format than that of 'gzip' in a web server. To test this, I created a 60k text only web page, and loaded it on to a number of different web servers. One by Yahoo, one by Google, one by 1and1.com, as well as a couple of others. They all compressed the file down to about 3.5 KB. The instances of Zlib compress the same file down to about 3KB, and are not recognized by browsers.

Re: Having some trouble compressing Web Pages

Posted: Sat Aug 10, 2013 8:47 am
by fizban
That's weird. Note that I have a site where I pre-compress the static css and js files in order to reduce the load of the server. To do that, I simply gzip those files and modify the .htaccess to redirect the requests to the css and js files to their css.gz and js.gz counterparts. I can suggest you test your compressed file the same way. If it works, the problem is not in the compression algorithm, but in a different part of the code...
my .htaccess looks like this:

Code: Select all

AddEncoding x-gzip .gz
RewriteEngine On
RewriteCond %{REQUEST_FILENAME}.gz -f
RewriteCond %{REQUEST_FILENAME} \.css$ [OR] 
RewriteCond %{REQUEST_FILENAME} \.js$
RewriteRule (.*)$ $1.gz
<FilesMatch "\\.css.gz$">
ForceType text/css
Header set Content-Encoding: gzip
</FilesMatch>

<FilesMatch "\\.js.gz$">
ForceType text/js
Header set Content-Encoding: gzip
</FilesMatch>

Re: Having some trouble compressing Web Pages

Posted: Sat Aug 10, 2013 5:40 pm
by RichAlgeni
What do you use as your web server Fizban? I have written my own server because of the circumstances. I tested the compressed file created by ZLib, and that created by mainstream web servers by creating a text only web page of 60KB, then I used a PB test 'GET' process to get the data served. I stripped off the header, and saved the remainder as a '.gz' file and was able to open it. I then took the uncompressed data, and compressed it with one of a number of ZLib libraries found in PureBasic or the forums.The ZLib compressed file was smaller than the '.gz' file, and was not recognized as a valid '.gz' file by winrar.

Do you have a GZip.exe process?

Re: Having some trouble compressing Web Pages

Posted: Sat Aug 10, 2013 8:18 pm
by fizban
I tipically use Apache. The XAMPP package is good for development and testing.

For the project I mentioned, I actually used zlibstat.lib importing the required functions:

Code: Select all

Import "ZlibStat.lib" 
  gzclose(a.l) As "_gzclose@4" 
  gzopen(a.l,b.l) As "_gzopen@8" 
  gzread(a.l,b.l,c.l) As "_gzread@12" 
EndImport 
I guess for writing it would be:

Code: Select all

gzwrite(a.l,b.l,c.l) As "_gzwrite@12" 
I am not sure which zlibstat.lib I used. Most likely I compiled it myself. I uploaded it here.

Re: Having some trouble compressing Web Pages

Posted: Sun Aug 11, 2013 2:50 am
by idle
just for reference a quick hack of atomic web server demo from 4.61
save the source where ever
create a dir under it as www
save a html page in it as index.html
run it
browse to 127.0.0.1:1081

as long as the accept-encoding request field says deflate you can use it

Code: Select all

;
; ------------------------------------------------------------
;
;       Atomic Web Server in PureBasic by AlphaSND
;
;           (c) 2001 - Fantaisie Software
;
; ------------------------------------------------------------
;
; 25/03/2001
;   Added path relative feature. It can be started everywhere
;   Added the window to kill it easely
;
; 19/03/2001
;   Added some new features (clients closed automatically)
;
; 17/03/2001
;   First version.
;
CompilerIf #PB_Compiler_OS = #PB_OS_Windows 
  
  ImportC "zlib.lib"
    compress2(*dest,*destlen,*source,sourcelen,level)
    uncompress(*dest,*destlen,*source,sourcelen)
  EndImport 
CompilerElse 
  
  ImportC "-lz"   
     compress(*dest,*destlen,*source,Srclen)
     compress2(*dest,*destlen,*source,sourcelen,level)
     uncompress(*dest,*destlen,*source,sourcelen)
 EndImport 
  
CompilerEndIf 



If InitNetwork() = 0
  MessageRequester("Error", "Can't initialize the network !", 0)
  End
EndIf

server=1
Port = 8081
BaseDirectory$ = "www/"
DefaultPage$   = "Index.html"
AtomicTitle$   = "Atomic Web Server v1.0"

Global EOL$

EOL$ = Chr(13)+Chr(10)

*Buffer = AllocateMemory(10000)

If CreateNetworkServer(1,Port)

  OpenWindow(0, 0, 0, 300, 0, "Atomic Web Server (Port "+Str(Port)+")",#PB_Window_SystemMenu | #PB_Window_ScreenCentered)

  Repeat

    Repeat
      WEvent = WindowEvent()

      If WEvent = #PB_Event_CloseWindow : Quit = 1 : EndIf
    Until WEvent = 0

    SEvent = NetworkServerEvent()

    If SEvent

      ClientID.l = EventClient();NetworkClientID()

      Select SEvent

        Case 1  ; When a new client has been connected...
          Debug "New connection"

        Case 4  ; When a client has closed the connection...
          Debug "Deconnection"

        Default
          Debug "Raw data"
          RequestLength.l = ReceiveNetworkData(ClientID, *Buffer, 2000)
         ; Debug PeekS(*buffer,RequestLength) 
          
          Gosub ProcessRequest

      EndSelect

    Else
      Delay(20)  ; Don't stole the whole CPU !
    EndIf

  Until Quit = 1

  CloseNetworkServer(1)
Else
  MessageRequester(AtomicTitle$, "Error: can't create the server (port in use ?).", 0)
EndIf

End



Procedure.l BuildRequestHeader(*Buffer, DataLength.l, ContentType$)

  Length = PokeS(*Buffer, "HTTP/1.1 200 OK"+EOL$)                     : *Buffer+Length
  Length = PokeS(*Buffer, "Date: Wed, 07 Aug 1996 11:15:43 GMT"+EOL$) : *Buffer+Length
  Length = PokeS(*Buffer, "Server: Atomic Web Server 0.2b"+EOL$)      : *Buffer+Length
  Length = PokeS(*Buffer, "Content-Length: "+Str(DataLength)+EOL$)    : *Buffer+Length
  Length = PokeS(*Buffer, "Content-Type: "+ContentType$+EOL$)         : *Buffer+Length
  Length = PokeS(*Buffer, "Content-Encoding: deflate" + EOL$) : *Buffer+Length 
  Length = PokeS(*Buffer, EOL$)                                       : *Buffer+Length
 
  ProcedureReturn *Buffer
EndProcedure


ProcessRequest:

  a$ = PeekS(*Buffer)

  If Left(a$, 3) = "GET"

    MaxPosition = FindString(a$, Chr(13), 5)
    Position = FindString(a$, " ", 5)
    If Position < MaxPosition
      RequestedFile$ = Mid(a$, 6, Position-5)      ; Automatically remove the leading '/'
      RequestedFile$ = RTrim(RequestedFile$)
    Else
      RequestedFile$ = Mid(a$, 6, MaxPosition-5)   ; When a command like 'GET /' is sent..
    EndIf

      ; The following routine transforme all '/' in '\' (Windows format)
      ;
      Structure tmp
        a.b
      EndStructure

      If RequestedFile$ = ""
        RequestedFile$ = DefaultPage$
      Else
        *t.tmp = @RequestedFile$
        While *t\a <> 0
          If *t\a = '/' : *t\a = '\' : EndIf
          *t+1
        Wend
      EndIf

      ; Test if the file exists, and if not display the error message
      ;

      If ReadFile(0, BaseDirectory$+RequestedFile$)

        FileLength = Lof(0)

        Select LCase(Right(RequestedFile$,4))
          Case ".gif"
            ContentType$ = "image/gif"

          Case ".jpg"
            ContentType$ = "image/jpeg"

          Case ".txt"
            ContentType$ = "text/plain"

          Case ".zip"
            ContentType$ = "application/zip"

          Default
            ContentType$ = "text/html"

        EndSelect
     
        
        *src = AllocateMemory(FileLength) 
        ReadData(0,*src,FileLength)
        *dest = AllocateMemory(FileLength*1.5) 
        destlen = FileLength*1.5 
        compress2(*dest,@destlen,*src,FileLength,9) 
               
        *FileBuffer = AllocateMemory(destlen+500)
        *BufferOffset = BuildRequestHeader(*FileBuffer, destlen, ContentType$)
        
        Debug PeekS(*FileBuffer,*BufferOffset-*FileBuffer)
        headerlen = *BufferOffset-*FileBuffer 
        
        CopyMemory(*dest,*FileBuffer+headerlen,destlen)
        FileLength = destlen + headerlen       
        Debug "header len " + headerlen
        Debug "compressed page  " + FileLength 
                
        CloseFile(0)

        SendNetworkData(ClientID, *FileBuffer, FileLength)
        FreeMemory(*FileBuffer)
        FreeMemory(*src)
        FreeMemory(*dest)
        
      Else
        If ReadFile(0, BaseDirectory$+"AtomicWebServer_Error.html")
          FileLength = Lof(0)
          ContentType$ = "text/html"

          *FileBuffer   = AllocateMemory(FileLength+200)
          *BufferOffset = BuildRequestHeader(*FileBuffer, FileLength, ContentType$)

          ReadData(0,*BufferOffset, FileLength)
          CloseFile(0)

          SendNetworkData(ClientID, *FileBuffer, *BufferOffset-*FileBuffer+FileLength)
          FreeMemory(*FileBuffer)
        EndIf
      EndIf
  EndIf

Return


Re: Having some trouble compressing Web Pages

Posted: Sun Aug 11, 2013 4:50 am
by PB

Code: Select all

Select Right(RequestedFile$,4)
This should really be:

Code: Select all

Select LCase(Right(RequestedFile$,4))
Because not all file extensions are lowercase.

Re: Having some trouble compressing Web Pages

Posted: Sun Aug 11, 2013 5:52 am
by idle
PB wrote:

Code: Select all

Select Right(RequestedFile$,4)
This should really be:

Code: Select all

Select LCase(Right(RequestedFile$,4))
Because not all file extensions are lowercase.
updated it thanks

Re: Having some trouble compressing Web Pages

Posted: Sun Aug 11, 2013 9:24 am
by fizban
Apparently, according to the zlib manual if you actually want to use gzip instead of deflate, you should call deflateInit2 with a specific value in the windowBits parameter

Code: Select all

#windowBits 15
#GZIP_ENCODING 16
#Z_DEFAULT_COMPRESSION  -1
#Z_DEFAULT_STRATEGY    0
#Z_DEFLATED   8
deflateInit2 (strm, #Z_DEFAULT_COMPRESSION, #Z_DEFLATED,
                             #windowBits | #GZIP_ENCODING, 8,
                             #Z_DEFAULT_STRATEGY)
and strm needs to be a z_stream. You can find a C example here

Re: Having some trouble compressing Web Pages

Posted: Mon Aug 12, 2013 3:05 am
by RichAlgeni
Perfect!

Thanks Idle!!!

I owe you one!

Re: Having some trouble compressing Web Pages

Posted: Thu Aug 29, 2013 1:08 am
by RichAlgeni
Idle is going to put a hit out on me after this, but here goes:

This whole issue came about because of data being lost from a sever to a client. Research has led me to believe this was the result of a 'router black hole.' It turned out that it was being caused by a old VPN product. We were able to bypass the problem by using 'Deflate', and code suggested by Idle.

For those not up to speed, this was a download of data into a web form, so that reports could be completed. The data is being sent out over a Motorola radio system, using a product called RadioIP.

Kludgy enough? That's what I thought! Then I asked them 'how do you get the filled out reports to the office?'

Ready for the answer?!?!

They print them out, and mail them in!!!

After all the work, research, and input from some great people, they use the mail to send in their reports!

Should have known!