How to compress with zlib

Just starting out? Need help? Post your questions and find answers here.
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: How to compress with zlib

Post by JHPJHP »

No it's not corrupt if it's producing a file with a matching Hash; well it is to your program... I'm guessing your program requires a certain type of compression. You can try modifing the WindowBits parameter - some possibilities:

Code: Select all

*Payload = DeflatePayload(*Input, 15 | #ENABLE_GZIP)

Code: Select all

*Payload = DeflatePayload(*Input, 15)

Code: Select all

*Payload = DeflatePayload(*Input, -15)
(change this for both *Payload & *Output)

I previously posted a link to the documentation which explains this in full.
- I also found this doing some general searches
-- 8..15 = zlib format
-- 24..31 = gzip format
-- -8..-15 = raw zlib format with no header


Also it may be caused by this line: LengthToWrite = DeflateBound(@strm, LengthToRead) / 20
- modify it without dividing, if that doesn't work then divide by 2, and increase incrementally for each new test

Good luck moob!
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

Congrats JHP! :D
JHPJHP wrote:No it's not corrupt if it's producing a file with a matching Hash; well it is to your program... I'm guessing your program requires a certain type of compression.
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

With the zlib and PB I can compress the original shared .dds file down to 81KB and still worky. But I had to manually keep altering LengthToWrite = * line and testing the compressed files.

You shouldn't need to be altering the deflateBound() returns unless its with +1 for PB maybe... The lib is spitting back lies or the way we are accepting is faulty. How I'm seeing it now, deflateBound() is pretty much useless in its current condition.
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: How to compress with zlib

Post by JHPJHP »

Ok, a whole new approach.

Thunder93 or skywalk can you confirm the following with me?..

Please download my DLL version of the code (a couple posts back). Next if you don't already have it (but I bet you do) download and install GIMP: http://www.gimp.org/. This next part you will probably need; download and copy the extracted dds plugin to the correct folder - here is the link: https://code.google.com/p/gimp-dds/ (readme.txt will give the location of the plugin folder).

Run the DLL version of the code (in its original state), then using GIMP open the original .dds file, then the compresed .dds file, "for the hell of it" open all three. :wink:
- this test shows that there is no problem opening/viewing the compressed .dds file

Moob if your reading this please do the same...
NB*: I sent an email to Mark Adler (co-creator of zlib) to clarify the use of deflateBound.
Cheers!
Last edited by JHPJHP on Mon Oct 21, 2013 8:06 pm, edited 2 times in total.
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

Yep. I have and use GIMP. :)

No need for this experiment. We know the file extracted is the same or it wouldn't share same hash.

And is apples to oranges regarding my experiences. :wink:
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: How to compress with zlib

Post by JHPJHP »

This test was not for the extracted .dds file but for the compressed one. I thought maybe a compressed .dds file had to be a certain compression ratio to be read - this proves that compression is not a factor.
- the compressed file opens in GIMP (with the addon) perfectly, just like the uncompressed one
Last edited by JHPJHP on Mon Oct 21, 2013 10:32 pm, edited 1 time in total.
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

Ahhhh, gotcha. Also perfect match between the two. :lol:
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

I don't know how it'll work on different dds files but working with the original dds uncompressed file that mood shared with us. I'm able to milk it down to 90KB with zlib lib.

Code: Select all

LengthToWrite = LengthToRead / 22.76 ;;;; Max Compression
And you know the compressed data is a match if the extracted data and the original data hashes are the same. :wink:
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

As I expected.. Even when the code works for the dds file that was shared, but when using different dds files we can have problems.

You know this line....

Code: Select all

LengthToWrite = deflateBound(@strm, LengthToRead) / 20
dds file I have gets messed up even by using..

Code: Select all

LengthToWrite = deflateBound(@strm, LengthToRead) / 2

We have to factor in for file size, header information if exists... I'm no good with math so I'll leave it up to someone else to work out. Hope you can get this straighten out moob. :?

Thunder93 wrote:

Code: Select all

LengthToWrite = deflateBound(@strm, LengthToRead) / 2
This isn't right.. It won't work well with larger files, perfectly fine for smaller ones.
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
moob
User
User
Posts: 68
Joined: Mon Aug 01, 2011 6:16 pm

Re: How to compress with zlib

Post by moob »

JHPJHP wrote:Ok, a whole new approach.

Thunder93 or skywalk can you confirm the following with me?..

Please download my DLL version of the code (a couple posts back). Next if you don't already have it (but I bet you do) download and install GIMP: http://www.gimp.org/. This next part you will probably need; download and copy the extracted dds plugin to the correct folder - here is the link: https://code.google.com/p/gimp-dds/ (readme.txt will give the location of the plugin folder).

Run the DLL version of the code (in its original state), then using GIMP open the original .dds file, then the compresed .dds file, "for the hell of it" open all three. :wink:
- this test shows that there is no problem opening/viewing the compressed .dds file

Moob if your reading this please do the same...
NB*: I sent an email to Mark Adler (co-creator of zlib) to clarify the use of deflateBound.
Cheers!
@JHPJHP

I have the photoshop with nvidia dds plug-in and I can't open the file is corrupt

the program i post before for compress and uncompress work great there is no problem with the program i post before

i can open in photoshop the compress file generated by that program.

but I can not open the dds compress file generate by PB 5.00 with the code here on my computer the file is corrupt
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

moob, just update the LengthToWrite line with the following. It should definitely work with deflateBound results only... :lol:

Code: Select all

LengthToWrite = deflateBound(@strm, LengthToRead)
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: How to compress with zlib

Post by JHPJHP »

Hi moob,

I can feel your frustration, my only point was that the file is not really corrupted if the Hash matches that of the original file - which you already confirmed. The Photoshop dds plugin may have a more strict check then the GIMP one, which is why both Thunder93 and myself can view the compressed file, and your attempts have failed. Again why I suggested testing with GIMP.

I don't believe this has anything to do with the code, beside what has been previously mentioned (including the fix), or what version of PureBasic you're using; which the DLL version of the script would have taken care of.

I wish we could have resolved this, but I think I've done all I can. Take care and good luck!
IdeasVacuum
Always Here
Always Here
Posts: 6425
Joined: Fri Oct 23, 2009 2:33 am
Location: Wales, UK
Contact:

Re: How to compress with zlib

Post by IdeasVacuum »

Luis has worked on zlib compression: http://www.purebasic.fr/english/viewtop ... 17&start=0 and has produced a wrapper for it. I have not tried it, but all of Luis's work is high quality.
IdeasVacuum
If it sounds simple, you have not grasped the complexity.
User avatar
Thunder93
Addict
Addict
Posts: 1788
Joined: Tue Mar 21, 2006 12:31 am
Location: Canada

Re: How to compress with zlib

Post by Thunder93 »

Hey JHPJHP. Did Mark ever responded to your email?
JHPJHP wrote:NB*: I sent an email to Mark Adler (co-creator of zlib) to clarify the use of deflateBound.
ʽʽSuccess is almost totally dependent upon drive and persistence. The extra energy required to make another effort or try another approach is the secret of winning.ʾʾ --Dennis Waitley
JHPJHP
Addict
Addict
Posts: 2129
Joined: Sat Oct 09, 2010 3:47 am
Contact:

Re: How to compress with zlib

Post by JHPJHP »

I didn't hear back, but after reading your post I followed up with another couple emails... the following is what was asked and answered:

Question
In the below code I’m passing a 2mb file for compression, and I thought by using deflateBound the optimized compression size would be returned. But the compressed file size is larger than the original file size; is this to be expected? And is there a correct way to determine the optimized compression size?
Code Included with Question

Code: Select all

Procedure DeflatePayload(*Input, windowBits.i)
  LengthToRead = MemorySize(*Input)
  strm.Z_STREAM
  strm\next_in = *Input
  strm\avail_in = LengthToRead
  strm\zalloc = #Z_NULL
  strm\zfree = #Z_NULL
  strm\opaque = #Z_NULL
  deflateInit2_(@strm, #Z_DEFAULT_COMPRESSION, #Z_DEFLATED, windowBits, #MAX_MEM_LEVEL, #Z_DEFAULT_STRATEGY, #ZLIB_VERSION, SizeOf(Z_STREAM))
  LengthToWrite = deflateBound(@strm, LengthToRead)
  *Payload = AllocateMemory(LengthToWrite)
  strm\next_out = *Payload
  strm\avail_out = LengthToWrite
  deflate(@strm, #Z_FINISH)
  deflateEnd(@strm)
  ProcedureReturn *Payload
EndProcedure
Response
deflateBound() bounds by how much the uncompressed data can possibly be expanded by compression. For example, if you feed random data to a compressor, the result will be larger. deflateBound() must give a number larger than the source length, since by definition a compressor must make some inputs larger. It is the worst case size of the result, allowing the user to allocate a buffer large enough to hold the result.
Question
... trying to figure out how to determine what to set for [ strm\avail_out ] when using [ deflateInit2_ ] in a single pass.
- set it to small and there is data loss
Response
That is what deflateBound() is for.
Additional Information from Website
deflateBound() returns an upper bound on the compressed size after deflation of sourceLen bytes. It must be called after deflateInit() or deflateInit2(). This would be used to allocate an output buffer for deflation in a single pass, and so would be called before deflate(). If that first deflate() call is provided the sourceLen input bytes, an output buffer allocated to the size returned by deflateBound(), and the flush value Z_FINISH, then deflate() is guaranteed to return Z_STREAM_END. Note that it is possible for the compressed size to be larger than the value returned by deflateBound() if flush options other than Z_FINISH or Z_NO_FLUSH are used.
I still don't know of a way to determine (on a single pass) the best size for output... My guess is not to do it in a single pass, but in a loop monitoring the responses.
Post Reply