PureBasic Forums - English

Here are some wrappers for the compression/decompression of memory buffers using the following compression libraries:

LZMA
aPLib
zlib
bzip2

for both 32/64 bits, Windows. It should be easy to use them under other OSES too, you should only need to compile the libs for your platform (libraries' source are not included in this zip).


Since each lib has its own "interface" I tried to make them more consistent in the wrappers, same thing for the return values.

Now if you can use one, you can use them all.

The functions are:

; In brief:
; compression: good ratio, very good speed, very low memory requirements (around 256 KB)
; decompression: extremely fast, very low memory requirements (should be less than 48 KB)

; In brief:
; compression: very good ratio, good speed, very customizable ram requirements (see dictionary size)
; decompression: very good speed, ram requirements depending on the dictionary size used for compression

; In brief:
; compression: good ratio, average speed, low memory requirements (around 640 KB)
; decompression: extremely fast, extremely low memory requirements

; In brief:
; compression: very good ratio, good speed, average memory requirements (from 1200 KB to 7600 KB)
; decompression: good speed, average memory requirements (same as compression)


*SourceBuff is the source buffer you want to compress/uncompress
*DestBuff is the destination buffer that will contain the compressed/uncompressed data
iSourceSize is the size in bytes of the data to be compressed/uncompressed
iDestSize is the size in bytes of the destination buffer

After the data has been compressed/decompressed, *iDestSize (it's a pointer to integer) will contain the actual length of the data compressed/uncompressed.

When compressing, if the size of source data can not be reduced, *DestSize will contain -1

To calculate the size of the buffer you should allocate as destination when compressing you can use the CalcDestBufferSize_*() helper procedure. It will return the size appropriate for each packer depending on the size of the source buffer.

When decompressing you can use GetOriginalBufferSize_*() to read back the size of the original uncompressed data. It's saved in a header at the start of the compressed buffer.

So ... to compress:
You allocate the space for the source data
You allocate the space for the destination data using CalcDestBufferSize_*()
You call PackMemory_*()
You test the return value and maybe *iDestSize
Now you have the compressed data in the destination buffer, the total size *iDestSize includes some bytes from the header (the header is specific to the packer used, usually it's simply a long).

To decompress:
You allocate the space for the source data
You get the destination space needed calling GetOriginalBufferSize_*() and allocate it
You call UnpackMemory_*()
You test the return value and maybe *iDestSize
Now you have the original uncompressed data, and its size is stored again for convenience in *iDestSize.

Keep in mind these are in-memory pack/unpack routines without CRC checks or any type of file structure. If you need them (and probably you will) you have to add what you need the way you like.

I've made a simple program as an example to pack/unpack a file.

I just finished all this and tested it a little, so if you find any error please let me know. Thanks.

Download [... from someone who got a copy at the time].

Bye!

PS: if you know some other opensource packer you feel could complement these without being redundant (it should offer something different) let me know !


EDIT (Feb 28): changed the test program and added some sample data to test against
EDIT (Mar 4) : changed the defaults to maximum compression for all to better compare the results

Results of the updated test program to give you the general idea:

Thanks, nice work!
Some suggestions for other packers: How about bzip2, lzo, ucl ?

Added bzip2 !

Great job Luis, these will come in very handy!

Thanks!

Hi guys, you are welcome.

I briefly tested UCL/LZO but the compression ratio seem quite low. The most similar seem to be aPLib but generally it compress better and has very low requirements for decompression too. Having that already in the collection it seem to me those two can be skipped.

bzip2 was really needed, but now probably the algorithms included here are enough for almost anything.

hey luis, I'd like to check this out but it seems your link to the zip in the first post in the thread is broken-- it keeps redirecting to http://www.newsguy.com/

Or do I need to have an account in order to get to that?

thanks!

Back online !

sweet!

it's working now

I love this!

Could be very useful. Thanks.

Would you explain what you did to compile to a static library? I get a 2kb file and a dependency to LzmaLib.dll (which doesn't exist) when I try to compile as static.

Am I missing a definition somewhere?

Mistrel wrote:Would you explain what you did to compile to a static library?

I don't remember the exact steps required, moreover I probably used a compiler different from yours... (VS 2005). I remember most of the original projects were targeting a dll. I went in and changed some settings to produce the static libs. In some cases (don't remember for which lib) I did some slightly mods to the original source for some reason, but I believe that was because I was trying to uniform the interfaces.

Anyway I've looked around my hd and found the modified projects I used at the time. I put the lzma project online here, have a look at it if you like. If you are using a MS C compiler you should find the settings you need to enable in your project for the compiler/linker/librarian.

Thank you. I will have a look at this.

I have a build environment set up for VC2005 so compiling should be a problem. I just need the correct options.

Thanks a lot..very useful

Link seems to be broken... Unless that's just me?