How to compress with zlib

Just starting out? Need help? Post your questions and find answers here.
User avatar
JHPJHP
Addict
Addict
Posts: 2251
Joined: Sat Oct 09, 2010 3:47 am

Re: How to compress with zlib

Post by JHPJHP »

The following code will work as is, but you should read the documentation on the zlib site, then optimize it to fit your needs.
- the current example tries to inflate / deflate all at once, but this may not be the best solution for your requirements
- as I mentioned in a previous post - if your file is not text based you would have to modify the code a bit
-- I suggest comparing the two examples to see how they work with the different formats
- notice that I set the compressed size with: LengthToWrite = deflateBound(@strm, LengthToRead) / 2
-- not sure why delatebound is not optimizing compression - thus the reason for "dividing by two" (modify to whatever works)

Code: Select all

#ENABLE_GZIP = 16
#Z_NULL = 0
#ZLIB_VERSION = "1.2.8"
#Z_FINISH = 4
#Z_DEFAULT_COMPRESSION = -1
#Z_DEFLATED = 8
#MAX_MEM_LEVEL = 9
#Z_DEFAULT_STRATEGY = 0

Structure Z_STREAM
  *next_in.Byte
  avail_in.l
  total_in.l
  *next_out.Byte
  avail_out.l
  total_out.l
  *msg.Byte
  *state
  zalloc.l
  zfree.l
  opaque.l
  data_type.i
  adler.l
  reserved.l
EndStructure

ImportC "zlib.lib"
  inflateInit2_(*strm, windowBits.i, Version.s, strm_size)
  inflate(*strm, flush.i)
  inflateEnd(*strm)
  deflateInit2_(*strm, level.i, method.i, windowBits.i, memlevel.i, strategy.i, Version.s, strm_size)
  deflateBound(*strm, sourceLen)
  deflate(*strm, flush.i)
  deflateEnd(*strm)
EndImport

Procedure InflatePayload(*Payload, windowBits.i)
  LengthToRead = MemorySize(*Payload)
  LengthToWrite = FileSize("test.dds")
  *Output = AllocateMemory(LengthToWrite)
  strm.Z_STREAM
  strm\next_in = *Payload
  strm\avail_in = LengthToRead
  strm\next_out = *Output
  strm\avail_out = LengthToWrite
  strm\zalloc = #Z_NULL
  strm\zfree = #Z_NULL
  strm\opaque = #Z_NULL
  inflateInit2_(@strm, windowBits, #ZLIB_VERSION, SizeOf(Z_STREAM))
  inflate(@strm, #Z_FINISH)
  inflateEnd(@strm)
  ProcedureReturn *Output
EndProcedure

Procedure DeflatePayload(*Input, windowBits.i)
  LengthToRead = MemorySize(*Input)
  strm.Z_STREAM
  strm\next_in = *Input
  strm\avail_in = LengthToRead
  strm\zalloc = #Z_NULL
  strm\zfree = #Z_NULL
  strm\opaque = #Z_NULL
  deflateInit2_(@strm, #Z_DEFAULT_COMPRESSION, #Z_DEFLATED, windowBits, #MAX_MEM_LEVEL, #Z_DEFAULT_STRATEGY, #ZLIB_VERSION, SizeOf(Z_STREAM))
  LengthToWrite = deflateBound(@strm, LengthToRead) / 2
  *Payload = AllocateMemory(LengthToWrite)
  strm\next_out = *Payload
  strm\avail_out = LengthToWrite
  deflate(@strm, #Z_FINISH)
  deflateEnd(@strm)
  ProcedureReturn *Payload
EndProcedure

If OpenFile(0, "test.dds")
  LengthToRead = FileSize("test.dds")
  *Input = AllocateMemory(LengthToRead)
  ReadData(0, *Input, LengthToRead)
  CloseFile(0)
  *Payload = DeflatePayload(*Input, 15 | #ENABLE_GZIP)

  If CreateFile(0, "test.gz")
    WriteData(0, *Payload, MemorySize(*Payload))
    CloseFile(0)
  EndIf
  FreeMemory(*Payload)
EndIf

If OpenFile(0, "test.gz")
  LengthToRead = FileSize("test.gz")
  *Payload = AllocateMemory(LengthToRead)
  ReadData(0, *Payload, LengthToRead)
  CloseFile(0)
  *Output = InflatePayload(*Payload, 15 | #ENABLE_GZIP)

  If CreateFile(0, "test2.dds")
    WriteData(0, *Output, MemorySize(*Output))
    CloseFile(0)
  EndIf
  FreeMemory(*Output)
  FreeMemory(*Payload)
EndIf
Last edited by JHPJHP on Sat Oct 19, 2013 4:18 am, edited 1 time in total.

If you're not investing in yourself, you're falling behind.

My PureBasic StuffFREE STUFF, Scripts & Programs.
My PureBasic Forum ➤ Questions, Requests & Comments.
moob
User
User
Posts: 68
Joined: Mon Aug 01, 2011 6:16 pm

Re: How to compress with zlib

Post by moob »

JHPJHP wrote:The following code will work as is, but you should read the documentation on zlib site, then optimize it to fit your needs.
- the current example tries to inflate / deflate all at once, but this may not be the best solution for your requirements
- as I mentioned in a previous post - if your file is not text based you would have to modify the code a bit
-- I suggest comparing the two examples to see how they work with the different formats
- notice that I set the compressed size with: LengthToWrite = deflateBound(@strm, LengthToRead) / 2
-- not sure why delatebound is not optimizing compression - thus the reason for "dividing by two" (modify to whatever works)

Code: Select all

#ENABLE_GZIP = 16
#Z_NULL = 0
#ZLIB_VERSION = "1.2.8"
#Z_FINISH = 4
#Z_DEFAULT_COMPRESSION = -1
#Z_DEFLATED = 8
#MAX_MEM_LEVEL = 9
#Z_DEFAULT_STRATEGY = 0

Structure Z_STREAM
  *next_in.Byte
  avail_in.l
  total_in.l
  *next_out.Byte
  avail_out.l
  total_out.l
  *msg.Byte
  *state
  zalloc.l
  zfree.l
  opaque.l
  data_type.i
  adler.l
  reserved.l
EndStructure

ImportC "zlib.lib"
  inflateInit2_(*strm, windowBits.i, Version.s, strm_size)
  inflate(*strm, flush.i)
  inflateEnd(*strm)
  deflateInit2_(*strm, level.i, method.i, windowBits.i, memlevel.i, strategy.i, Version.s, strm_size)
  deflateBound(*strm, sourceLen)
  deflate(*strm, flush.i)
  deflateEnd(*strm)
EndImport

Procedure InflatePayload(*Payload, windowBits.i)
  LengthToRead = MemorySize(*Payload)
  LengthToWrite = FileSize("test.dds")
  *Output = AllocateMemory(LengthToWrite)
  strm.Z_STREAM
  strm\next_in = *Payload
  strm\avail_in = LengthToRead
  strm\next_out = *Output
  strm\avail_out = LengthToWrite
  strm\zalloc = #Z_NULL
  strm\zfree = #Z_NULL
  strm\opaque = #Z_NULL
  inflateInit2_(@strm, windowBits, #ZLIB_VERSION, SizeOf(Z_STREAM))
  inflate(@strm, #Z_FINISH)
  inflateEnd(@strm)
  ProcedureReturn *Output
EndProcedure

Procedure DeflatePayload(*Input, windowBits.i)
  LengthToRead = MemorySize(*Input)
  strm.Z_STREAM
  strm\next_in = *Input
  strm\avail_in = LengthToRead
  strm\zalloc = #Z_NULL
  strm\zfree = #Z_NULL
  strm\opaque = #Z_NULL
  deflateInit2_(@strm, #Z_DEFAULT_COMPRESSION, #Z_DEFLATED, windowBits, #MAX_MEM_LEVEL, #Z_DEFAULT_STRATEGY, #ZLIB_VERSION, SizeOf(Z_STREAM))
  LengthToWrite = deflateBound(@strm, LengthToRead) / 2
  *Payload = AllocateMemory(LengthToWrite)
  strm\next_out = *Payload
  strm\avail_out = LengthToWrite
  deflate(@strm, #Z_FINISH)
  deflateEnd(@strm)
  ProcedureReturn *Payload
EndProcedure

If OpenFile(0, "test.dds")
  LengthToRead = FileSize("test.dds")
  *Input = AllocateMemory(LengthToRead)
  ReadData(0, *Input, LengthToRead)
  CloseFile(0)
  *Payload = DeflatePayload(*Input, 15 | #ENABLE_GZIP)

  If CreateFile(0, "test.gz")
    WriteData(0, *Payload, MemorySize(*Payload))
    CloseFile(0)
  EndIf
  FreeMemory(*Payload)
EndIf

If OpenFile(0, "test.gz")
  LengthToRead = FileSize("test.gz")
  *Payload = AllocateMemory(LengthToRead)
  ReadData(0, *Payload, LengthToRead)
  CloseFile(0)
  *Output = InflatePayload(*Payload, 15 | #ENABLE_GZIP)

  If CreateFile(0, "test2.dds")
    WriteData(0, *Output, MemorySize(*Output))
    CloseFile(0)
  EndIf
  FreeMemory(*Output)
  FreeMemory(*Payload)
EndIf
In the last example debug give an error in line 75 "*Input = AllocateMemory(LengthToRead)", the error is "Can't allocate a memory block of size 0"

thanks for your effort

but I need compress dds file and not txt and no one of your example work in dds file.

You have done much trying help me thanks for that, I'm a noob in this.

thank you
User avatar
JHPJHP
Addict
Addict
Posts: 2251
Joined: Sat Oct 09, 2010 3:47 am

Re: How to compress with zlib

Post by JHPJHP »

NB*: These are only examples, you will need to modify the code to fit your existing script.

Double check the file names:

Code: Select all

LengthToRead = FileSize("test.dds")
*Input = AllocateMemory(LengthToRead)
LengthToRead is the file size of your .dds file, if your getting "Can't allocate a memory block of size 0" then your .dds file is size 0, or you're not pointing to the correct file.

Test Only - hard code the value:

Code: Select all

LengthToRead = 2097280 ;size of the file you supplied in your post
*Input = AllocateMemory(LengthToRead)

If you're not investing in yourself, you're falling behind.

My PureBasic StuffFREE STUFF, Scripts & Programs.
My PureBasic Forum ➤ Questions, Requests & Comments.
User avatar
skywalk
Addict
Addict
Posts: 4211
Joined: Wed Dec 23, 2009 10:14 pm
Location: Boston, MA

Re: How to compress with zlib

Post by skywalk »

Hi moob,
Please don't reply with full quotes of previous post. :idea:
Also, break your problem down. If you cannot compress/uncompress small amounts of data, don't bother with your large dds file.
A text file is still just binary data. Of course, text will compress more than some random binary data, but that is not the point.
The nice thing about standards is there are so many to choose from. ~ Andrew Tanenbaum
User avatar
JHPJHP
Addict
Addict
Posts: 2251
Joined: Sat Oct 09, 2010 3:47 am

Re: How to compress with zlib

Post by JHPJHP »

A couple good points skywalk, I should have mentioned earlier about "full quotes".

You probably haven't read through all the posts (and who could blame you), but moob has already compressed and decompressed a small text file; the new scripts I gave him are for binary data... I think he's just making some noob / moob mistakes. :P

The scripts you provided earlier would have worked just as well. :wink:
Last edited by JHPJHP on Sat Oct 19, 2013 3:03 am, edited 1 time in total.

If you're not investing in yourself, you're falling behind.

My PureBasic StuffFREE STUFF, Scripts & Programs.
My PureBasic Forum ➤ Questions, Requests & Comments.
User avatar
skywalk
Addict
Addict
Posts: 4211
Joined: Wed Dec 23, 2009 10:14 pm
Location: Boston, MA

Re: How to compress with zlib

Post by skywalk »

Haha, yes I applaud your patience so far.
I got confused by moob's questions since I am compressing/uncompressing all day long with my backup tool.
File sizes or types don't matter at least for the method I posted.
The nice thing about standards is there are so many to choose from. ~ Andrew Tanenbaum
User avatar
JHPJHP
Addict
Addict
Posts: 2251
Joined: Sat Oct 09, 2010 3:47 am

Re: How to compress with zlib

Post by JHPJHP »

Hi moob,

I did another test using the .dds file you supplied - increasing the compression. I modified the division from 2 to 20... as I mentioned in previous posts - change this value to fit your needs.

Code: Select all

LengthToWrite = deflateBound(@strm, LengthToRead) / 20
I tested the output file against your original file using UltraCompare, and the files were identical. If you going to increase the compression using this method, be sure to check for data loss.

NB*: I sent an email to Mark Adler (co-creator of zlib) to clarify the use of deflateBound.

Cheers!

If you're not investing in yourself, you're falling behind.

My PureBasic StuffFREE STUFF, Scripts & Programs.
My PureBasic Forum ➤ Questions, Requests & Comments.
moob
User
User
Posts: 68
Joined: Mon Aug 01, 2011 6:16 pm

Re: How to compress with zlib

Post by moob »

JHPJHP wrote:Hi moob,

I did another test using the .dds file you supplied - increasing the compression. I modified the division from 2 to 20... as I mentioned in previous posts - change this value to fit your needs.

Code: Select all

LengthToWrite = deflateBound(@strm, LengthToRead) / 20
I tested the output file against your original file using UltraCompare, and the files were identical. If you going to increase the compression using this method, be sure to check for data loss.

NB*: I sent an email to Mark Adler (co-creator of zlib) to clarify the use of deflateBound.

Cheers!

Ok thanks I can compress, but there is something wrong with compression, the file is corrupt after compressed.


This is what inside of my test.dds without compression.

Image


And this is what inside after compression

Image

as you see there is nothing inside after compression.

the program used can open dds file compressed with zlib

And I have trying uncompress the file with other program and the result is this, the same result.

Image


I can compress with your example and the example of skywalk

but the result is an corrupt file.

I can't uncompress with external program the compressed file with examples presented by you and skywalk here, because the result is an corrupt dds file.

english is not my first language, I hope you both can understand what I write and not misunderstood it.

and yes I'm noob in PB, I not understand many code you write here because of this I ask for help.
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

Hi moob. Revisit JHPJHP post - http://www.purebasic.fr/english/viewtop ... 84#p428484 and re-copy the code. Now REM out the old Z_STREAM structure and use the following...

Code: Select all

Structure Z_STREAM Align #PB_Structure_AlignC
  *next_in.Byte
  avail_in.l
  total_in.l
  *next_out.Byte
  avail_out.l
  total_out.l
  *msg.Byte
  *state
  zalloc.l
  zfree.l
  opaque.l
  data_type.i
  adler.l
  reserved.l
  CompilerIf #PB_Compiler_Processor = #PB_Processor_x64
    alignment.l
  CompilerEndIf
EndStructure
I did reproduce your experiences with the corruption, and this update will address it. :D
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

8e8c5f619da1de67d67480671fc2827c test.dds -- Original
b6c1f472b21d542185e4e4a78a569b12 test.gz -- Created
8e8c5f619da1de67d67480671fc2827c test2.dds -- Extracted

And my extraction w/another decompressor of the test.gz file
8e8c5f619da1de67d67480671fc2827c test

And doesn't matter if you using 32bit or 64bit compiler. :)
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
User avatar
JHPJHP
Addict
Addict
Posts: 2251
Joined: Sat Oct 09, 2010 3:47 am

Re: How to compress with zlib

Post by JHPJHP »

Hi Thunder93,

I gave moob the same Structure layout that your suggesting, but that caused a different set of problems (see previous posts)... the modified Structure was only to test with, as was the compression ratio... but definitely worth revisiting - good call. :)

Hi moob,

Try testing the compressed .gz file from http://2zip.org/ (site: thanks to Thunder93) - see if that makes a difference?
- please note that all the examples and suggestions have been tested before hand - and found working
-- this may indicate an incorrect setting under "Compiler Options" or elsewhere
Last edited by JHPJHP on Sun Oct 20, 2013 4:36 am, edited 1 time in total.

If you're not investing in yourself, you're falling behind.

My PureBasic StuffFREE STUFF, Scripts & Programs.
My PureBasic Forum ➤ Questions, Requests & Comments.
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

'Align' keyword support for structure was introduced w/Version 5.10 of PB and moob using 5.00. I can't personally imagine not updating already or installing the later versions along side of that current version.

If moob is coding under x64 system, you'll have to have two source files one with and another without the structure paddings.
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
moob
User
User
Posts: 68
Joined: Mon Aug 01, 2011 6:16 pm

Re: How to compress with zlib

Post by moob »

Thunder93 wrote:'Align' keyword support for structure was introduced w/Version 5.10 of PB and moob using 5.00. I can't personally imagine not updating already or installing the later versions along side of that current version.

If moob is coding under x64 system, you'll have to have two source files one with and another without the structure paddings.

I'm stuck in PB 5.00 because I need to use the DevIL lib for convert png to dds and only found an wrapper working in version 5.00

And I don't know how to update the code of wrapper to work in PB 5.20.

I'm stuck in version 5.00 because of this and 'Align' keyword not work in PB 5.00
JHPJHP wrote:Hi Thunder93,

I gave moob the same Structure layout that your suggesting, but that caused a different set of problems (see previous posts)... the modified Structure was only to test with, as was the compression ratio... but definitely worth revisiting - good call. :)

Hi moob,

Try testing the compressed .gz file from http://2zip.org/ (site: thanks to Thunder93) - see if that makes a difference?
- please note that all the examples and suggestions have been tested before hand - and found working
-- this may indicate an incorrect setting under "Compiler Options" or elsewhere
No diference, the size is the same 116.81KB
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

Thanks moob. I overlooked you mentioning that earlier.

Replace the previous structure for 5.00 with this one, and try your luck :wink:

Code: Select all

Structure Z_STREAM
  *next_in.Byte
  avail_in.l
  total_in.l
  *next_out.Byte
  avail_out.l
  total_out.l
  *msg.Byte
  *state
  zalloc.l     
  zfree.l  
  opaque.l
  PB_Alignment1.b[4]
  data_type.i
  adler.l
  reserved.l
  PB_Alignment2.b[8]
EndStructure
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
User avatar
JHPJHP
Addict
Addict
Posts: 2251
Joined: Sat Oct 09, 2010 3:47 am

Re: How to compress with zlib

Post by JHPJHP »

Hi Thunder93,

The file size being returned from moob's tests seem to indicate a separate problem?
Last edited by JHPJHP on Mon Oct 21, 2013 6:10 am, edited 9 times in total.

If you're not investing in yourself, you're falling behind.

My PureBasic StuffFREE STUFF, Scripts & Programs.
My PureBasic Forum ➤ Questions, Requests & Comments.
Post Reply