Page 1 of 1
UseLZ4Packer()
Posted: Tue Jul 26, 2016 4:23 am
by nco2k
when it comes to speed, lz4 is probably the best choice. it would be really cool to have it.
c ya,
nco2k
Re: UseLZ4Packer()
Posted: Tue Jul 26, 2016 6:08 am
by wilbert
nco2k wrote:when it comes to speed, lz4 is probably the best choice. it would be really cool to have it.
I never heard of it but the source codes are very small
http://www.lz4.org
Unfortunately the PureBasic SDK doesn't contain any information about the Packer library otherwise it might have been possible to add your own compression algorithms.
Re: UseLZ4Packer()
Posted: Tue Jul 26, 2016 6:54 am
by Keya
nco2k wrote:when it comes to speed, lz4 is probably the best choice.
but
is it?!?
it has BSD license which is good, and i have no doubt it is fast, but my google for
"faster than LZ4" yielded quite a few good looking results heehee
btw see also
Comparison of PB's native packers, because really there are
5 main factors to look at when considering any packer, decompression speed being one
(actually make that 6 - good license lol)
That comparison shows that the FASTEST (unpacker) is ZIP, which we all know is good but not great at this metric, so there's room for improvement by another packer there.
It also shows that the ONLY SMALL unpacker is BriefLZ at ~10kb - everything else = 200kb, so also room for
smaller packers/unpackers. You may not gain much space from compressing parts of your app if your decompressor is 200kb.
My impression/guess also from that is that the packer libraries are currently just one .a file with both pack+unpack code as the only .o file within, rather than multiple .o files (pack.o, unpack.o etc) in each lib, so for example the whole library is being included rather than just the decompression code even if you don't use compression.
Re: UseLZ4Packer()
Posted: Tue Jul 26, 2016 8:01 am
by wilbert
I have the impression that the decompression speed depends a lot of what kind of data is being compressed / decompressed.
On OSX El Capitan, you can use LZ4 and sometimes it seems to be very fast and other times a bit slower.
In case someone has OSX El Capitan and wants to try, these are the functions you can use to test.
Code: Select all
; OSX 10.11+
#COMPRESSION_LZ4 = $100
#COMPRESSION_ZLIB = $205
#COMPRESSION_LZMA = $306
#COMPRESSION_LZ4_RAW = $101
#COMPRESSION_LZFSE = $801
ImportC "-lcompression"
compression_encode_buffer(*dst_buffer, dst_size, *src_buffer, src_size, *scratch_buffer, algorithm)
compression_decode_buffer(*dst_buffer, dst_size, *src_buffer, src_size, *scratch_buffer, algorithm)
EndImport
*buf1 = AllocateMemory(99999)
*buf2 = AllocateMemory(99999)
*buf3 = AllocateMemory(99999)
csize = compression_encode_buffer(*buf2, 99999, *buf1, 99999, #Null, #COMPRESSION_LZ4)
usize = compression_decode_buffer(*buf3, 99999, *buf2, csize, #Null, #COMPRESSION_LZ4)
Debug csize
Debug usize
In general I think it might be interesting for PureBasic as it seems to outperform BriefLZ (when it comes to decompression speed) and is tiny.
Re: UseLZ4Packer()
Posted: Wed Jul 27, 2016 7:43 am
by tj1010
Isn't LZMA2 the the most balanced compression based on most benchmarks? If you just want speed go with deflate which is already built in to most systems.
Re: UseLZ4Packer()
Posted: Wed Jul 27, 2016 8:01 am
by Keya
i dont know about LZMA2 specifically but LZMA is super strong compression ratio but comes at the price of being very slow for both compression & decompression. But LZMA2 doesn't seem to be much of a change to LZMA's actual compression, it seems more like structural changes to support things like uncompressed data also for more flexibility (i could be wrong, just my impression from reading)
Re: UseLZ4Packer()
Posted: Wed Jul 27, 2016 11:06 am
by tj1010
Keya wrote:i dont know about LZMA2 specifically but LZMA is super strong compression ratio but comes at the price of being very slow for both compression & decompression. But LZMA2 doesn't seem to be much of a change to LZMA's actual compression, it seems more like structural changes to support things like uncompressed data also for more flexibility (i could be wrong, just my impression from reading)
It tops ratio in benchmark but yeah is cpu and ram consuming. For cheap hardware I'd use deflate or even just a 64MB string-replace binary header in a quad separated binary file using a low-priority delayed thread splitting where needed and with page/swap support.