Wrappers for some compression libraries (x86/x64)

[luis]

Here are some wrappers for the compression/decompression of memory buffers using the following compression libraries:

LZMA
aPLib
zlib
bzip2

for both 32/64 bits, Windows. It should be easy to use them under other OSES too, you should only need to compile the libs for your platform (libraries' source are not included in this zip).


Since each lib has its own "interface" I tried to make them more consistent in the wrappers, same thing for the return values.

Now if you can use one, you can use them all.

The functions are:

Code: Select all

Procedure.i PackMemory_ZLIB (*SourceBuff, *DestBuff, iSourceSize, *iDestSize, iCompressionLevel = 9)
Procedure.i UnpackMemory_ZLIB (*SourceBuff, *DestBuff, iSourceSize, *iDestSize)
Procedure.i GetOriginalBufferSize_ZLIB (*SourceBuff)
Procedure.i CalcDestBufferSize_ZLIB (iSourceBuffSize)

; In brief:
; compression: good ratio, very good speed, very low memory requirements (around 256 KB)
; decompression: extremely fast, very low memory requirements (should be less than 48 KB)

Code: Select all

Procedure.i PackMemory_LZMA (*SourceBuff, *DestBuff, iSourceSize, *iDestSize, iCompressionLevel = 9, iDictSize = #LZMA_DICT_4M)
Procedure.i UnpackMemory_LZMA (*SourceBuff, *DestBuff, iSourceSize, *iDestSize)
Procedure.i CalcDestBufferSize_LZMA (SourceBuffSize)
Procedure.i GetOriginalBufferSize_LZMA (*SourceBuff)

; In brief:
; compression: very good ratio, good speed, very customizable ram requirements (see dictionary size)
; decompression: very good speed, ram requirements depending on the dictionary size used for compression

Code: Select all

Procedure.i PackMemory_APLIB (*SourceBuff, *DestBuff, iSourceSize, *iDestSize)
Procedure.i UnpackMemory_APLIB (*SourceBuff, *DestBuff, iSourceSize, *iDestSize)
Procedure.i CalcDestBufferSize_APLIB (iSourceBuffSize)
Procedure.i GetOriginalBufferSize_APLIB (*SourceBuff)

; In brief:
; compression: good ratio, average speed, low memory requirements (around 640 KB)
; decompression: extremely fast, extremely low memory requirements

Code: Select all

Procedure.i PackMemory_BZIP2 (*SourceBuff, *DestBuff, iSourceSize, *iDestSize, iBlockSize = 9)
Procedure.i UnpackMemory_BZIP2 (*SourceBuff, *DestBuff, iSourceSize, *iDestSize)
Procedure.i CalcDestBufferSize_BZIP2 (iSourceBuffSize)
Procedure.i GetOriginalBufferSize_BZIP2 (*SourceBuff)

; In brief:
; compression: very good ratio, good speed, average memory requirements (from 1200 KB to 7600 KB)
; decompression: good speed, average memory requirements (same as compression)


*SourceBuff is the source buffer you want to compress/uncompress
*DestBuff is the destination buffer that will contain the compressed/uncompressed data
iSourceSize is the size in bytes of the data to be compressed/uncompressed
iDestSize is the size in bytes of the destination buffer

After the data has been compressed/decompressed, *iDestSize (it's a pointer to integer) will contain the actual length of the data compressed/uncompressed.

When compressing, if the size of source data can not be reduced, *DestSize will contain -1

To calculate the size of the buffer you should allocate as destination when compressing you can use the CalcDestBufferSize_*() helper procedure. It will return the size appropriate for each packer depending on the size of the source buffer.

When decompressing you can use GetOriginalBufferSize_*() to read back the size of the original uncompressed data. It's saved in a header at the start of the compressed buffer.

So ... to compress:
You allocate the space for the source data
You allocate the space for the destination data using CalcDestBufferSize_*()
You call PackMemory_*()
You test the return value and maybe *iDestSize
Now you have the compressed data in the destination buffer, the total size *iDestSize includes some bytes from the header (the header is specific to the packer used, usually it's simply a long).

To decompress:
You allocate the space for the source data
You get the destination space needed calling GetOriginalBufferSize_*() and allocate it
You call UnpackMemory_*()
You test the return value and maybe *iDestSize
Now you have the original uncompressed data, and its size is stored again for convenience in *iDestSize.

Keep in mind these are in-memory pack/unpack routines without CRC checks or any type of file structure. If you need them (and probably you will) you have to add what you need the way you like.

I've made a simple program as an example to pack/unpack a file.

I just finished all this and tested it a little, so if you find any error please let me know. Thanks.

Download [... from someone who got a copy at the time].

Bye!

PS: if you know some other opensource packer you feel could complement these without being redundant (it should offer something different) let me know !


EDIT (Feb 28): changed the test program and added some sample data to test against
EDIT (Mar 4) : changed the defaults to maximum compression for all to better compare the results

Results of the updated test program to give you the general idea:

Code: Select all

LZM
Packing data\source\texture.bmp from 786486 to 531066 (67.5%) in 369 ms.
Unpacking from 531066 to 786486 in 68 ms.
Packing data\source\ascii.txt from 1346274 to 385564 (28.6%) in 907 ms.
Unpacking from 385564 to 1346274 in 57 ms.
Packing data\source\win32pe.exe from 908248 to 135441 (14.9%) in 210 ms.
Unpacking from 135441 to 908248 in 39 ms.
Packing data\source\source.c from 256052 to 58827 (23.0%) in 108 ms.
Unpacking from 58827 to 256052 in 16 ms.
Packing data\source\pdfdata.pdf from 499601 to 440328 (88.1%) in 178 ms.
Unpacking from 440328 to 499601 in 53 ms.

APL
Packing data\source\texture.bmp from 786486 to 660246 (83.9%) in 1296 ms.
Unpacking from 660246 to 786486 in 30 ms.
Packing data\source\ascii.txt from 1346274 to 532640 (39.6%) in 6104 ms.
Unpacking from 532640 to 1346274 in 43 ms.
Packing data\source\win32pe.exe from 908248 to 317222 (34.9%) in 639 ms.
Unpacking from 317222 to 908248 in 28 ms.
Packing data\source\source.c from 256052 to 69152 (27.0%) in 442 ms.
Unpacking from 69152 to 256052 in 13 ms.
Packing data\source\pdfdata.pdf from 499601 to 483512 (96.8%) in 147 ms.
Unpacking from 483512 to 499601 in 20 ms.

ZLI
Packing data\source\texture.bmp from 786486 to 570737 (72.6%) in 97 ms.
Unpacking from 570737 to 786486 in 26 ms.
Packing data\source\ascii.txt from 1346274 to 488408 (36.3%) in 237 ms.
Unpacking from 488408 to 1346274 in 37 ms.
Packing data\source\win32pe.exe from 908248 to 314205 (34.6%) in 174 ms.
Unpacking from 314205 to 908248 in 28 ms.
Packing data\source\source.c from 256052 to 66276 (25.9%) in 46 ms.
Unpacking from 66276 to 256052 in 14 ms.
Packing data\source\pdfdata.pdf from 499601 to 451755 (90.4%) in 54 ms.
Unpacking from 451755 to 499601 in 20 ms.

BZ2
Packing data\source\texture.bmp from 786486 to 497305 (63.2%) in 246 ms.
Unpacking from 497305 to 786486 in 133 ms.
Packing data\source\ascii.txt from 1346274 to 348949 (25.9%) in 374 ms.
Unpacking from 348949 to 1346274 in 168 ms.
Packing data\source\win32pe.exe from 908248 to 191452 (21.1%) in 856 ms.
Unpacking from 191452 to 908248 in 75 ms.
Packing data\source\source.c from 256052 to 58635 (22.9%) in 58 ms.
Unpacking from 58635 to 256052 in 25 ms.
Packing data\source\pdfdata.pdf from 499601 to 454343 (90.9%) in 281 ms.
Unpacking from 454343 to 499601 in 94 ms.

[javabean]

Thanks, nice work!
Some suggestions for other packers: How about bzip2, lzo, ucl ?

[luis]

Added bzip2 !

[Inf0Byt3]

Great job Luis, these will come in very handy!

Thanks!

[luis]

Hi guys, you are welcome.

I briefly tested UCL/LZO but the compression ratio seem quite low. The most similar seem to be aPLib but generally it compress better and has very low requirements for decompression too. Having that already in the collection it seem to me those two can be skipped.

bzip2 was really needed, but now probably the algorithms included here are enough for almost anything.

[jamba]

hey luis, I'd like to check this out but it seems your link to the zip in the first post in the thread is broken-- it keeps redirecting to http://www.newsguy.com/

Or do I need to have an account in order to get to that?

thanks!

[luis]

Back online !

[jamba]

sweet!

it's working now

[Joakim Christiansen]

I love this!

[srod]

Could be very useful. Thanks.

[Mistrel]

Would you explain what you did to compile to a static library? I get a 2kb file and a dependency to LzmaLib.dll (which doesn't exist) when I try to compile as static.

Am I missing a definition somewhere?

[luis]

Would you explain what you did to compile to a static library?

I don't remember the exact steps required, moreover I probably used a compiler different from yours... (VS 2005). I remember most of the original projects were targeting a dll. I went in and changed some settings to produce the static libs. In some cases (don't remember for which lib) I did some slightly mods to the original source for some reason, but I believe that was because I was trying to uniform the interfaces.

Anyway I've looked around my hd and found the modified projects I used at the time. I put the lzma project online here, have a look at it if you like. If you are using a MS C compiler you should find the settings you need to enable in your project for the compiler/linker/librarian.

[Mistrel]

Thank you. I will have a look at this.

I have a build environment set up for VC2005 so compiling should be a problem. I just need the correct options.

[Kwai chang caine]

Thanks a lot..very useful

[Arcturus]

Link seems to be broken... Unless that's just me?