Comparison of PB's native packers

Share your advanced PureBasic knowledge/code with the community.
User avatar
Keya
Addict
Addict
Posts: 1890
Joined: Thu Jun 04, 2015 7:10 am

Comparison of PB's native packers

Post by Keya »

Purebasic spoils us with 5 different built-in compression codecs (6 if you count the uncompressed bundling of TAR, or 7 if you count the deprecated x86-only JCALG1)
These are: ZIP, LZMA, BriefLZ, TAR.GZip, and TAR.BZip2 (ps. Fred it would be nice if BZIP2 and GZIP were available on their own, rather than having to use TAR bundling! TAR is ewww but BZIP2 is awesome!)

So here's a simple test program to help find out which of Purebasic's built-in packers is best suited to the data we want to compress/decompress.

The variables are PACK TIME, UNPACK TIME, PACK SIZE/RATIO, and PB EXE SIZE. Each packer of course has its own strengths and weaknesses!

Here's the results of a 91 megabyte SQL dump (UTF8 textfile). Tested with Purebasic x86 5.40 LTS in Windows XP. I havent had a chance to do x64 test.
I ran each test twice and only recorded the 2nd result to ensure 'even caching':

Code: Select all

                ZIP        LZMA       BriefLZ    TAR        TAR.GZ     TAR.BZ2       "Winning" order (TAR excluded):
                --------------------------------------------------------------       ----------------
PACK TIME       3175ms     67647ms    1653ms     1141ms     3325ms     9464ms        BriefLZ, ZIP, TAR.GZ, TAR.BZ2, LZMA
UNPACK TIME     1532ms     2254ms     2273ms     1252ms     1332ms     3145ms        TAR.GZ, ZIP, LZMA, BriefLZ, TAR.BZ2
PACK SIZE       25218579   18887882   34247532   94030848   25218522   19782588      
PACK RATIO      26.820%    20.087%    36.422%    100.002%   26.820%    21.039%       LZMA, TAR.BZ2, TAR.GZ, ZIP, BriefLZ
PB.EXE SIZE     207,872    291,328    15,872     219,648    219,648    219,648       BriefLZ, ZIP, TAR.GZ/TAR.BZ2, LZMA
So, from this one test with this dataset, we can summarize:
ZIP: A great balance of compression and speed.
LZMA: Extremely slow compression, but thats the price you pay for the very best compression ratio.
BriefLZ: Extremely good performance in both packing & unpacking, and the resulting exe is still TINY unlike the others. 36% compression to ZIPs 26% means it's not great, but still pretty good.
TAR.GZ: On par with ZIP.
TAR.BZ2: Compression ratio so very close to LZMA yet compression speed is 7x faster. Decompression 1.4x slower than LZMA however.

Quite a nice menu to choose from, depending on ours projects individual needs :)
One of the nice surprises I found was that the BriefLZ codec only adds 10kb to the exe - thats with full compress + decompress support! And its speed of both is very good, especially compression but still respectable decompress speed. Its compression ratio is also quite respectable too, compressing my 91mb sample file to 36%. Compared to ZIPs 27% its not great but still pretty good, and overall I think it makes an excellent packer for executable resources!

Code: Select all

#PACKER = 1  ;1=ZIP  2=LZMA  3=BriefLZ  4=TAR  5=TAR.GZ  6=TAR.BZ2

sFileToPack.s = "c:\temp\91mb.txt"            ;Test file to compress. Ensure you have write access to c:\temp\
sFileToSave.s = "c:\temp\mytemparchive"       ;Save to this archive file (minus extension)

CompilerIf #PB_Compiler_Debugger = 1
  MessageRequester("Disable the debugger first", "Error": End   ;in future ill use CompilerError
CompilerEndIf


CompilerIf #PACKER = 1
  UseZipPacker()
  sSaveFile.s = sFileToSave + ".zip"
  PackerId = #PB_PackerPlugin_Zip
  sResult.s = "Packer: ZIP" + #CRLF$
CompilerElseIf #PACKER = 2
  UseLZMAPacker()
  sSaveFile.s = sFileToSave + ".lzma"
  PackerId = #PB_PackerPlugin_Lzma
  sResult.s = "Packer: LZMA" + #CRLF$
CompilerElseIf #PACKER = 3
  UseBriefLZPacker()
  sSaveFile.s = sFileToSave + ".brieflz"
  PackerId = #PB_PackerPlugin_BriefLZ
  sResult.s = "Packer: BriefLZ" + #CRLF$
CompilerElseIf #PACKER = 4
  UseTARPacker()
  sSaveFile.s = sFileToSave + ".tar"
  PackerId = #PB_PackerPlugin_Tar  
  sResult.s = "Packer: TAR" + #CRLF$
CompilerElseIf #PACKER = 5
  UseTARPacker()
  sSaveFile.s = sFileToSave + ".tar.gz"
  PackerId = #PB_PackerPlugin_Tar | #PB_Packer_Gzip   
  sResult.s = "Packer: TAR.GZ" + #CRLF$
CompilerElseIf #PACKER = 6
  UseTARPacker()
  sSaveFile.s = sFileToSave + ".tar.bz2"
  PackerId = #PB_PackerPlugin_Tar | #PB_Packer_Bzip2   
  sResult.s = "Packer: TAR.BZ2" + #CRLF$
CompilerElse
  MessageRequester("Invalid packer id","Error"): End
CompilerEndIf


DeleteFile(sSaveFile)
OrigSize.i = FileSize(sFileToPack)
hPack.i = CreatePack(#PB_Any, sSaveFile, PackerId)
If hPack = 0
  MessageRequester("Error","CreatePack failed")
Else
  Time1.i = GetTickCount_()
  AddPackFile(hPack, sFileToPack, "packed.dat")  
  TimePack.i = GetTickCount_() - Time1
  ClosePack(hPack)  
    
  hPack.i = OpenPack(#PB_Any, sSaveFile, PackerId) 
  If hPack = 0
    MessageRequester("Error","OpenPack failed")
  Else    
    DeleteFile("c:\~extract.dat")
    Time1.i = GetTickCount_()
    Result = ExaminePack(hPack)
    NextPackEntry(hPack)
    Unpacked.i = UncompressPackFile(hPack, "c:\~extract.dat")
    TimeUnpack.i = GetTickCount_() - Time1
    ClosePack(hPack)
  EndIf
  
  sResult = sResult + "Packed in " + Str(TimePack) + "ms" + #CRLF$ + 
            "Unpacked in " + Str(TimeUnpack) + "ms" + #CRLF$ + 
            "Packsize=" + Str(FileSize(sSaveFile)) + " (" + StrF((FileSize(sSaveFile)/ OrigSize) * 100,3) + "%)" + #CRLF$ +
            "Origsize=" + Str(OrigSize) + "  Unpacked=" + Str(Unpacked) + #CRLF$ + 
            "Executable size=" + Str(FileSize(ProgramFilename()))
  MessageRequester("Result", sResult)
  SetClipboardText(sResult)  
EndIf
Last edited by Keya on Tue Sep 01, 2015 9:04 am, edited 11 times in total.
Fred
Administrator
Administrator
Posts: 18243
Joined: Fri May 17, 2002 4:39 pm
Location: France
Contact:

Re: Comparison of PB's native packers

Post by Fred »

Nice one. I will move it to trick and tips. BTW, you can use 'CompilerError' instead of a syntax error to stop the compilation :P
User avatar
Keya
Addict
Addict
Posts: 1890
Joined: Thu Jun 04, 2015 7:10 am

Re: Comparison of PB's native packers

Post by Keya »

Fred seeing as BriefLZ is the only packer with a small footprint - all others are 200-300kb - it would be great if you add aPLib. It's open source, free/open license, plus 32bit and 64bit libs, dlls etc, for PE, ELF, COFF, and its TIIIINY, and its rock solid as its been around since 1998 and still updated in 2014. I know you'd be at least somewhat familiar with it.

For comparison it took 119 seconds to compress my 91mb textfile; nearly twice as slow as LZMA. But that's the only downside, and plenty of people are happy to wait for LZMA.
Compression ratio was down to 29.8%, or 7% smaller than BriefLZ, and only 3% larger than ZIP.
And, decompression time was just 499ms!!! Over 4x faster than BriefLZ, and 3x faster than ZIP/TAR.GZ which are the current PB fastest besides TAR.
And, decompression code footprint is just 169 bytes!!! 9.8kb smaller than BriefLZ's 10kb :D

http://ibsensoftware.com/products_aPLib.html
http://ibsensoftware.com/files/aPLib-1.1.1.zip
He also has a BriefLZ but its further down his list, im not sure if thats the one youre using in Purebasic? :)
wilbert
PureBasic Expert
PureBasic Expert
Posts: 3942
Joined: Sun Aug 08, 2004 5:21 am
Location: Netherlands

Re: Comparison of PB's native packers

Post by wilbert »

Keya wrote:Purebasic spoils us with 5 different built-in compression codecs (6 if you count the uncompressed bundling of TAR, or 7 if you count the deprecated x86-only JCALG1)
These are: ZIP, LZMA, BriefLZ, TAR.GZip, and TAR.BZip2 (ps. Fred it would be nice if BZIP2 and GZIP were available on their own, rather than having to use TAR bundling! TAR is ewww but BZIP2 is awesome!)

So here's a simple test program to help find out which of Purebasic's built-in packers is best suited to the data we want to compress/decompress.
Nice comparison.

Maybe also a comparison of StringFingerprint() and FileFingerprint() with all available algorithms 5.40 offers ? :wink:
I know it has nothing to do with packers but it would be nice to see their speed also compared on different platforms.
Windows (x64)
Raspberry Pi OS (Arm64)
User avatar
oakvalley
User
User
Posts: 77
Joined: Sun Aug 08, 2004 6:34 pm
Location: Norway
Contact:

Re: Comparison of PB's native packers

Post by oakvalley »

Is it just me or why doesn't PB5.40 LTS (with the new packers includes) also applies to CompressMemory() command?

PB5.40 manual says:
#PB_PackerPlugin_BriefLZ: use the BriefLZ packer to compress the memory. UseBriefLZPacker() has to be called to have any effect.
#PB_PackerPlugin_Zip : use the Zip packer to compress the memory. UseZipPacker() has to be called to have any effect.
#PB_PackerPlugin_LZMA : use the LZMA packer to compress the memory. UseLZMAPacker() has to be called to have any effect.

while CreatePack() says:
#PB_PackerPlugin_BriefLZ: use the BriefLZ packer to create the pack file. UseBriefLZPacker() has to be called to register the plugin.
#PB_PackerPlugin_Zip : use the Zip packer to create the pack file. UseZipPacker() has to be called to register the plugin.
#PB_PackerPlugin_Lzma : use the Lzma packer to create the pack file (also known as 7z archive). UseLZMAPacker() has to be called to register the plugin.
#PB_PackerPlugin_Tar : use the Tar packer to create the pack file. UseTarPacker() has to be called to register the plugin.
It can be combined with #PB_Packer_Gzip or #PB_Packer_Bzip2 to create compressed tar archive.

I don't see why TAR cannot be used to compress memory?
Regards Stone Oakvalley
Currently @ PB 5.70
infratec
Always Here
Always Here
Posts: 7620
Joined: Sun Sep 07, 2008 12:45 pm
Location: Germany

Re: Comparison of PB's native packers

Post by infratec »

That's simple:

tar is no compression.
Tar is only an archive format for storing many files in one file.
Since you have no files in memory and there is no compression by tar, it makes no sense to use tar on memory.

Bernd
User avatar
Keya
Addict
Addict
Posts: 1890
Joined: Thu Jun 04, 2015 7:10 am

Re: Comparison of PB's native packers

Post by Keya »

it's still unclear how Tar survived the mass extinction of the dinosaurs. Linux fanatics are an odd bunch
User avatar
oakvalley
User
User
Posts: 77
Joined: Sun Aug 08, 2004 6:34 pm
Location: Norway
Contact:

Re: Comparison of PB's native packers

Post by oakvalley »

However, I do mean TAR with GZip flag combined actually.

Look up the command for UseTarPacker() in PB5.40 manual
"UseTarPacker()"

[i]Description
Enable Tar compress, uncompress and Tar archive support to the packer library. Bzip2 and Gzip compression are both supported. Compressing and uncompressing Tar archives are usually fast.[/i]

It do say COMPRESS, not to mention, "see also OpenPack(), CompressMemory(), UncompressMemory()"

Makes no sense to see "CompressMemory()" when its either forgotten in the manual or it does not apply to combine UseTarPacker() with Compress/Uncompress memory to me.
Regards Stone Oakvalley
Currently @ PB 5.70
infratec
Always Here
Always Here
Posts: 7620
Joined: Sun Sep 07, 2008 12:45 pm
Location: Germany

Re: Comparison of PB's native packers

Post by infratec »

Keya wrote:it's still unclear how Tar survived the mass extinction of the dinosaurs. Linux fanatics are an odd bunch
It is totally clear:
Have you ever packed a symbolic link or a device with something else then tar :?:
Are the unix rights are preserved by any other packer/archiver :?:

That's why tar is still needed.

Not for windows, but for unix.

Bernd
infratec
Always Here
Always Here
Posts: 7620
Joined: Sun Sep 07, 2008 12:45 pm
Location: Germany

Re: Comparison of PB's native packers

Post by infratec »

@oakvalley

you are right, the helpfile is not clear about tar.
You can post it as bug in Documentation.

Bernd
User avatar
oakvalley
User
User
Posts: 77
Joined: Sun Aug 08, 2004 6:34 pm
Location: Norway
Contact:

Re: Comparison of PB's native packers

Post by oakvalley »

Ahh. Ok, so TAR only applicable for Linux users. I can live with that.
But, yes, minor confusion bug in documentation. Thanks for the clarification.
Regards Stone Oakvalley
Currently @ PB 5.70
User avatar
Keya
Addict
Addict
Posts: 1890
Joined: Thu Jun 04, 2015 7:10 am

Re: Comparison of PB's native packers

Post by Keya »

Until today PB's packers have been fixed in the middle - a good general-purpose balance between speed and compression ratio.

TODAY THE DOORS OPENED WIDER!!! Fred added LEVEL support (PB 5.60) to functions like CompressMemory() ... and it turns out ZIP, LZMA and BriefLZ all support it! We now have not 1 but 10 levels to choose from :)

Here's just a simple test compressing a 21mb bmp.

Code: Select all

DisableDebugger
UseZipPacker(): UseLZMAPacker(): UseBriefLZPacker()

hFile = ReadFile(#PB_Any, "c:\huge.bmp") ;21,108,056
flen = Lof(hFile)
*buf = AllocateMemory(flen)
ReadData(hFile, *buf, flen)
CloseFile(hFile)

#PB_Packer = #PB_PackerPlugin_Zip
;#PB_Packer = #PB_PackerPlugin_Lzma
;#PB_Packer = #PB_PackerPlugin_BriefLZ
OpenConsole()

*pack = AllocateMemory(flen)
For level = 0 To 9
  Time1=ElapsedMilliseconds()
  size = CompressMemory(*buf, flen, *pack, flen, #PB_Packer, level)
  Time2=ElapsedMilliseconds()
  PrintN("Level" + Str(level) + #TAB$ + RSet(Str(size),10," ") + LSet(StrF((size/flen)*100,3)+"%",10," ") + Str(Time2-Time1)+"ms")
Next level
Input()
=

Code: Select all

      was 21108056  100.0%
ZIP - the balanced lifestyle!
; Level0  0         0.000%    62ms
; Level1  11839612  56.090%   404ms
; Level2  11654860  55.215%   421ms
; Level3  11505051  54.505%   437ms
; Level4  11144497  52.797%   490ms
; Level5  11041823  52.311%   565ms
; Level6  10995763  52.093%   609ms
; Level7  10983598  52.035%   644ms
; Level8  10972013  51.980%   717ms
; Level9  10970843  51.975%   754ms

LZMA - every last bit!
; Level0   8691940  41.178%   1271ms
; Level1   8863068  41.989%   1418ms
; Level2   8944556  42.375%   1561ms
; Level3   8953213  42.416%   2064ms
; Level4   8928468  42.299%   3072ms
; Level5   7459434  35.339%   7609ms
; Level6   7459362  35.339%   7409ms
; Level7   7375828  34.943%   8348ms
; Level8   7375828  34.943%   8445ms
; Level9   7375828  34.943%   8424ms

BriefLZ - just get it done quick!
; Level0  15138296  71.718%   243ms
; Level1  13829442  65.517%   229ms
; Level2  13382651  63.401%   233ms
; Level3  13191942  62.497%   240ms
; Level4  13074596  61.941%   249ms
; Level5  12867456  60.960%   270ms
; Level6  12734563  60.330%   296ms
; Level7  12684851  60.095%   303ms
; Level8  12661453  59.984%   323ms
; Level9  12661453  59.984%   316ms
Dine-in or take-away? :)
FRED 2020
Post Reply