Accelerate LZMA compression

Everything else that doesn't fall into one of the other PB categories.
User avatar
jacdelad
Addict
Addict
Posts: 2010
Joined: Wed Feb 03, 2021 12:46 pm
Location: Riesa

Accelerate LZMA compression

Post by jacdelad »

Hi,
I have a program which stores a lot of text. To save space (it's really a lot of text) I use LZMA compression to reduce the size significantly. Now, compression so much text takes a lot of time and 7-Zip is much faster when using the same option (at least, what I expect to be the same options...).
Without discussing that I could use BriefLZ or split the data or whatever for the sacrifice of some space and other levers I could pull: Can the compression be sped up? I know that the built-in libraries have some more functions that are not mapped into PureBasic directly and from time to time some question is answered by importing this or that function and such. My program is using only one core when compressing, 7-Zip uses all available, so is this maybe already doable?
Good morning, that's a nice tnetennba!

PureBasic 6.21/Windows 11 x64/Ryzen 7900X/32GB RAM/3TB SSD
Synology DS1821+/DX517, 130.9TB+50.8TB+2TB SSD
Thorium
Addict
Addict
Posts: 1305
Joined: Sat Aug 15, 2009 6:59 pm

Re: Accelerate LZMA compression

Post by Thorium »

I think what you want to use is LZMA2 instead. It's basically the same as LZMA but allows multi-threaded compression. I don't know if this is already included in what comes with PureBasic.
User avatar
jacdelad
Addict
Addict
Posts: 2010
Joined: Wed Feb 03, 2021 12:46 pm
Location: Riesa

Re: Accelerate LZMA compression

Post by jacdelad »

It isn't. Thanks for the explanation, I didn't know that.
Good morning, that's a nice tnetennba!

PureBasic 6.21/Windows 11 x64/Ryzen 7900X/32GB RAM/3TB SSD
Synology DS1821+/DX517, 130.9TB+50.8TB+2TB SSD
User avatar
jacdelad
Addict
Addict
Posts: 2010
Joined: Wed Feb 03, 2021 12:46 pm
Location: Riesa

Re: Accelerate LZMA compression

Post by jacdelad »

Ok, so I found out that it's not the compression, it's fast enough. The real slowdown is done by ComposeJSON. I want to save a lost containing a structure (which includes basically every data type) with 70000 elements. Is there a better way than jsonizing it? Can I compress the JSON object directly? I'm not so firm with JSON, is it like a memory block, using JSONObjectSize() to get the size?
Good morning, that's a nice tnetennba!

PureBasic 6.21/Windows 11 x64/Ryzen 7900X/32GB RAM/3TB SSD
Synology DS1821+/DX517, 130.9TB+50.8TB+2TB SSD
Post Reply