Page 1 of 1

Accelerate LZMA compression

Posted: Fri Jul 05, 2024 12:38 pm
by jacdelad
Hi,
I have a program which stores a lot of text. To save space (it's really a lot of text) I use LZMA compression to reduce the size significantly. Now, compression so much text takes a lot of time and 7-Zip is much faster when using the same option (at least, what I expect to be the same options...).
Without discussing that I could use BriefLZ or split the data or whatever for the sacrifice of some space and other levers I could pull: Can the compression be sped up? I know that the built-in libraries have some more functions that are not mapped into PureBasic directly and from time to time some question is answered by importing this or that function and such. My program is using only one core when compressing, 7-Zip uses all available, so is this maybe already doable?

Re: Accelerate LZMA compression

Posted: Fri Jul 12, 2024 2:06 pm
by Thorium
I think what you want to use is LZMA2 instead. It's basically the same as LZMA but allows multi-threaded compression. I don't know if this is already included in what comes with PureBasic.

Re: Accelerate LZMA compression

Posted: Fri Jul 12, 2024 2:10 pm
by jacdelad
It isn't. Thanks for the explanation, I didn't know that.

Re: Accelerate LZMA compression

Posted: Sun Jul 14, 2024 8:47 am
by jacdelad
Ok, so I found out that it's not the compression, it's fast enough. The real slowdown is done by ComposeJSON. I want to save a lost containing a structure (which includes basically every data type) with 70000 elements. Is there a better way than jsonizing it? Can I compress the JSON object directly? I'm not so firm with JSON, is it like a memory block, using JSONObjectSize() to get the size?