Page 1 of 2
					
				Having some trouble compressing Web Pages
				Posted: Wed Aug 07, 2013 4:38 pm
				by RichAlgeni
				I thought that ZLib was a derivative of GZip, so I used that first. I also tried the included #PB_Packer_Zip CompressMemory command as Deflate. Both compressed the page, but neither were recognized by a browser.
Has anyone worked with this successfully?
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Wed Aug 07, 2013 4:56 pm
				by JHPJHP
				Found these posts - is this what your looking for?
(second post is informative - but not sure it still applies)
http://www.purebasic.fr/english/viewtopic.php?t=37984
viewtopic.php?t=36620 
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Wed Aug 07, 2013 6:18 pm
				by RichAlgeni
				Not sure. I used the zlib_64.lib import and invaluable source from Luis. The zlib is doing a great job at compressing. I'm adding 'Content-Encoding: gzip' to the header as specified, but the compressed data sent after the header is not recognized.
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Fri Aug 09, 2013 11:44 pm
				by fizban
				In one of my projects, I had to do the reverse, that is, to uncompress an gzipped http content. The easiest way I found was to create a temporary file, use the gzread function and delete the temporary file. I imagine that in order to compress the stream you could use the same technique, but using gzwrite.
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Sat Aug 10, 2013 12:56 am
				by RichAlgeni
				I will check that out, thanks @fizban!
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Sat Aug 10, 2013 2:21 am
				by RichAlgeni
				Just for FYI, I've used the zlib.lib included with PureBasic, as well as zlib_32.lib and zlib_64.lib from the forum. None of these libraries create a compressed file that is recognizable by browsers. They all compress files into a smaller format than that of 'gzip' in a web server. To test this, I created a 60k text only web page, and loaded it on to a number of different web servers. One by Yahoo, one by Google, one by 1and1.com, as well as a couple of others. They all compressed the file down to about 3.5 KB. The instances of Zlib compress the same file down to about 3KB, and are not recognized by browsers.
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Sat Aug 10, 2013 8:47 am
				by fizban
				That's weird. Note that I have a site where I pre-compress the static css and js files in order to reduce the load of the server. To do that, I simply gzip those files and modify the .htaccess to redirect the requests to the css and js files to their css.gz and js.gz counterparts. I can suggest you test your compressed file the same way. If it works, the problem is not in the compression algorithm, but in a different part of the code...
my .htaccess looks like this:
Code: Select all
AddEncoding x-gzip .gz
RewriteEngine On
RewriteCond %{REQUEST_FILENAME}.gz -f
RewriteCond %{REQUEST_FILENAME} \.css$ [OR] 
RewriteCond %{REQUEST_FILENAME} \.js$
RewriteRule (.*)$ $1.gz
<FilesMatch "\\.css.gz$">
ForceType text/css
Header set Content-Encoding: gzip
</FilesMatch>
<FilesMatch "\\.js.gz$">
ForceType text/js
Header set Content-Encoding: gzip
</FilesMatch>
 
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Sat Aug 10, 2013 5:40 pm
				by RichAlgeni
				What do you use as your web server Fizban? I have written my own server because of the circumstances. I tested the compressed file created by ZLib, and that created by mainstream web servers by creating a text only web page of 60KB, then I used a PB test 'GET' process to get the data served. I stripped off the header, and saved the remainder as a '.gz' file and was able to open it. I then took the uncompressed data, and compressed it with one of a number of ZLib libraries found in PureBasic or the forums.The ZLib compressed file was smaller than the '.gz' file, and was not recognized as a valid '.gz' file by winrar.
Do you have a GZip.exe process?
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Sat Aug 10, 2013 8:18 pm
				by fizban
				I tipically use Apache. The XAMPP package is good for development and testing. 
For the project I mentioned, I actually used zlibstat.lib importing the required functions:
Code: Select all
Import "ZlibStat.lib" 
  gzclose(a.l) As "_gzclose@4" 
  gzopen(a.l,b.l) As "_gzopen@8" 
  gzread(a.l,b.l,c.l) As "_gzread@12" 
EndImport 
I guess for writing it would be:
Code: Select all
gzwrite(a.l,b.l,c.l) As "_gzwrite@12" 
I am not sure which zlibstat.lib I used. Most likely I compiled it myself. I uploaded it 
here.
 
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Sun Aug 11, 2013 2:50 am
				by idle
				just for reference a quick hack of atomic web server demo from 4.61
save the source where ever
create a dir under it as www 
save a html page in it as index.html 
run it
browse to 127.0.0.1:1081 
as long as the accept-encoding request field says deflate you can use it 
 
Code: Select all
;
; ------------------------------------------------------------
;
;       Atomic Web Server in PureBasic by AlphaSND
;
;           (c) 2001 - Fantaisie Software
;
; ------------------------------------------------------------
;
; 25/03/2001
;   Added path relative feature. It can be started everywhere
;   Added the window to kill it easely
;
; 19/03/2001
;   Added some new features (clients closed automatically)
;
; 17/03/2001
;   First version.
;
CompilerIf #PB_Compiler_OS = #PB_OS_Windows 
  
  ImportC "zlib.lib"
    compress2(*dest,*destlen,*source,sourcelen,level)
    uncompress(*dest,*destlen,*source,sourcelen)
  EndImport 
CompilerElse 
  
  ImportC "-lz"   
     compress(*dest,*destlen,*source,Srclen)
     compress2(*dest,*destlen,*source,sourcelen,level)
     uncompress(*dest,*destlen,*source,sourcelen)
 EndImport 
  
CompilerEndIf 
If InitNetwork() = 0
  MessageRequester("Error", "Can't initialize the network !", 0)
  End
EndIf
server=1
Port = 8081
BaseDirectory$ = "www/"
DefaultPage$   = "Index.html"
AtomicTitle$   = "Atomic Web Server v1.0"
Global EOL$
EOL$ = Chr(13)+Chr(10)
*Buffer = AllocateMemory(10000)
If CreateNetworkServer(1,Port)
  OpenWindow(0, 0, 0, 300, 0, "Atomic Web Server (Port "+Str(Port)+")",#PB_Window_SystemMenu | #PB_Window_ScreenCentered)
  Repeat
    Repeat
      WEvent = WindowEvent()
      If WEvent = #PB_Event_CloseWindow : Quit = 1 : EndIf
    Until WEvent = 0
    SEvent = NetworkServerEvent()
    If SEvent
      ClientID.l = EventClient();NetworkClientID()
      Select SEvent
        Case 1  ; When a new client has been connected...
          Debug "New connection"
        Case 4  ; When a client has closed the connection...
          Debug "Deconnection"
        Default
          Debug "Raw data"
          RequestLength.l = ReceiveNetworkData(ClientID, *Buffer, 2000)
         ; Debug PeekS(*buffer,RequestLength) 
          
          Gosub ProcessRequest
      EndSelect
    Else
      Delay(20)  ; Don't stole the whole CPU !
    EndIf
  Until Quit = 1
  CloseNetworkServer(1)
Else
  MessageRequester(AtomicTitle$, "Error: can't create the server (port in use ?).", 0)
EndIf
End
Procedure.l BuildRequestHeader(*Buffer, DataLength.l, ContentType$)
  Length = PokeS(*Buffer, "HTTP/1.1 200 OK"+EOL$)                     : *Buffer+Length
  Length = PokeS(*Buffer, "Date: Wed, 07 Aug 1996 11:15:43 GMT"+EOL$) : *Buffer+Length
  Length = PokeS(*Buffer, "Server: Atomic Web Server 0.2b"+EOL$)      : *Buffer+Length
  Length = PokeS(*Buffer, "Content-Length: "+Str(DataLength)+EOL$)    : *Buffer+Length
  Length = PokeS(*Buffer, "Content-Type: "+ContentType$+EOL$)         : *Buffer+Length
  Length = PokeS(*Buffer, "Content-Encoding: deflate" + EOL$) : *Buffer+Length 
  Length = PokeS(*Buffer, EOL$)                                       : *Buffer+Length
 
  ProcedureReturn *Buffer
EndProcedure
ProcessRequest:
  a$ = PeekS(*Buffer)
  If Left(a$, 3) = "GET"
    MaxPosition = FindString(a$, Chr(13), 5)
    Position = FindString(a$, " ", 5)
    If Position < MaxPosition
      RequestedFile$ = Mid(a$, 6, Position-5)      ; Automatically remove the leading '/'
      RequestedFile$ = RTrim(RequestedFile$)
    Else
      RequestedFile$ = Mid(a$, 6, MaxPosition-5)   ; When a command like 'GET /' is sent..
    EndIf
      ; The following routine transforme all '/' in '\' (Windows format)
      ;
      Structure tmp
        a.b
      EndStructure
      If RequestedFile$ = ""
        RequestedFile$ = DefaultPage$
      Else
        *t.tmp = @RequestedFile$
        While *t\a <> 0
          If *t\a = '/' : *t\a = '\' : EndIf
          *t+1
        Wend
      EndIf
      ; Test if the file exists, and if not display the error message
      ;
      If ReadFile(0, BaseDirectory$+RequestedFile$)
        FileLength = Lof(0)
        Select LCase(Right(RequestedFile$,4))
          Case ".gif"
            ContentType$ = "image/gif"
          Case ".jpg"
            ContentType$ = "image/jpeg"
          Case ".txt"
            ContentType$ = "text/plain"
          Case ".zip"
            ContentType$ = "application/zip"
          Default
            ContentType$ = "text/html"
        EndSelect
     
        
        *src = AllocateMemory(FileLength) 
        ReadData(0,*src,FileLength)
        *dest = AllocateMemory(FileLength*1.5) 
        destlen = FileLength*1.5 
        compress2(*dest,@destlen,*src,FileLength,9) 
               
        *FileBuffer = AllocateMemory(destlen+500)
        *BufferOffset = BuildRequestHeader(*FileBuffer, destlen, ContentType$)
        
        Debug PeekS(*FileBuffer,*BufferOffset-*FileBuffer)
        headerlen = *BufferOffset-*FileBuffer 
        
        CopyMemory(*dest,*FileBuffer+headerlen,destlen)
        FileLength = destlen + headerlen       
        Debug "header len " + headerlen
        Debug "compressed page  " + FileLength 
                
        CloseFile(0)
        SendNetworkData(ClientID, *FileBuffer, FileLength)
        FreeMemory(*FileBuffer)
        FreeMemory(*src)
        FreeMemory(*dest)
        
      Else
        If ReadFile(0, BaseDirectory$+"AtomicWebServer_Error.html")
          FileLength = Lof(0)
          ContentType$ = "text/html"
          *FileBuffer   = AllocateMemory(FileLength+200)
          *BufferOffset = BuildRequestHeader(*FileBuffer, FileLength, ContentType$)
          ReadData(0,*BufferOffset, FileLength)
          CloseFile(0)
          SendNetworkData(ClientID, *FileBuffer, *BufferOffset-*FileBuffer+FileLength)
          FreeMemory(*FileBuffer)
        EndIf
      EndIf
  EndIf
Return
 
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Sun Aug 11, 2013 4:50 am
				by PB
				
This should really be:
Code: Select all
Select LCase(Right(RequestedFile$,4))
Because not all file extensions are lowercase.
 
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Sun Aug 11, 2013 5:52 am
				by idle
				PB wrote:
This should really be:
Code: Select all
Select LCase(Right(RequestedFile$,4))
Because not all file extensions are lowercase.
 
updated it thanks
 
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Sun Aug 11, 2013 9:24 am
				by fizban
				Apparently, according to the 
zlib manual if you actually want to use gzip instead of deflate, you should call deflateInit2 with a specific value in the windowBits parameter 
Code: Select all
#windowBits 15
#GZIP_ENCODING 16
#Z_DEFAULT_COMPRESSION  -1
#Z_DEFAULT_STRATEGY    0
#Z_DEFLATED   8
deflateInit2 (strm, #Z_DEFAULT_COMPRESSION, #Z_DEFLATED,
                             #windowBits | #GZIP_ENCODING, 8,
                             #Z_DEFAULT_STRATEGY)
and strm needs to be a z_stream. You can find 
a C example here 
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Mon Aug 12, 2013 3:05 am
				by RichAlgeni
				Perfect!
Thanks Idle!!!
I owe you one!
			 
			
					
				Re: Having some trouble compressing Web Pages
				Posted: Thu Aug 29, 2013 1:08 am
				by RichAlgeni
				Idle is going to put a hit out on me after this, but here goes:
This whole issue came about because of data being lost from a sever to a client. Research has led me to believe this was the result of a 'router black hole.' It turned out that it was being caused by a old VPN product. We were able to bypass the problem by using 'Deflate', and code suggested by Idle.
For those not up to speed, this was a download of data into a web form, so that reports could be completed. The data is being sent out over a Motorola radio system, using a product called RadioIP. 
Kludgy enough? That's what I thought! Then I asked them 'how do you get the filled out reports to the office?'
Ready for the answer?!?!
They print them out, and mail them in!!!
After all the work, research, and input from some great people, they use the mail to send in their reports!
Should have known!