Page 1 of 1
lzma unpack speed, extremely slow on big Archiv
Posted: Sun Apr 16, 2023 9:02 pm
by Taz
When I unpack a large archive with Purebasic, it takes a very long time. (There is only one file in the archive)
Here is a comparison with the console version of 7zr.exe
Code: Select all
FileSize: 538 MiB, unpack time:
PB: 18230 ms
7zr: 4215 ms
PureBasic is 333% slower
FileSize: 1076 MiB, unpack time:
PB: 31396 ms
7zr: 4921 ms
PureBasic is 538% slower
FileSize: 2128 MiB, unpack time:
PB: 67244 ms
7zr: 6961 ms
PureBasic is 866% slower
Here is an example code:
You can download a 1GB packed (538MB) test file
HERE. (the packed file contains only hex characters)
The required 7zr.exe
HERE, or directly from the
manufacturer.
Code: Select all
EnableExplicit
UseLZMAPacker()
Procedure.q _unpackPB(Archiv.s)
Protected File.s, pt.q = ElapsedMilliseconds()
If OpenPack(0, Archiv, #PB_PackerPlugin_Lzma) And ExaminePack(0)
While NextPackEntry(0)
If PackEntryType(0) = #PB_Packer_File
File = GetTemporaryDirectory() + PackEntryName(0)
UncompressPackFile(0, File)
Break
EndIf
Wend
ClosePack(0)
EndIf
pt = ElapsedMilliseconds() - pt
If FileSize(File)
DeleteFile(File)
EndIf
ProcedureReturn pt
EndProcedure
Procedure.q _unpack7z(Archiv.s)
Protected Dir.s = GetTemporaryDirectory() + "~bin", pt.q = ElapsedMilliseconds()
RunProgram("7zr.exe", ReplaceString("x -aoa '"+Archiv+"' -o'"+Dir+"'", "'", #DQUOTE$), "", #PB_Program_Hide|#PB_Program_Wait)
pt = ElapsedMilliseconds() - pt
If FileSize(Dir) = -2
DeleteDirectory(Dir, "")
EndIf
ProcedureReturn pt
EndProcedure
Structure tTEST
File.s
PureBasic.q
SevenZip.q
EndStructure
If FileSize("7zr.exe") = -1
MessageRequester("", "pls, download 7zr.exe first!")
End
EndIf
Define Str.s, Diff.d, NewList Test.tTEST()
AddElement(Test()): Test()\File = "bin.7z"
ForEach Test(): With Test()
If FileSize(\File) > 0
\PureBasic = _unpackPB(\File)
\SevenZip = _unpack7z(\File)
Str = "FileSize: " + Str(FileSize(\File)/1024/1024) + ~" MiB, unpack time:\n" +
" PB: " + RSet(Str(\PureBasic), 5)+ ~" ms\n" +
" 7zr: " + RSet(Str(\SevenZip), 5) + ~" ms\n"
Diff = \PureBasic / \SevenZip * 100 - 100
If Diff > 0
Str + "PureBasic is " + StrD(Diff, 0) + ~"% slower\n"
Else
Str + "PureBasis is faster!\n"
EndIf
Else
Str = "file '" + \File + "' not found!"
EndIf
CompilerIf #PB_Compiler_Debugger
Debug Str
CompilerElse
MessageRequester("Unpack speed", Str)
CompilerEndIf
EndWith:Next
Re: lzma unpack speed, extremely slow on big Archiv
Posted: Mon Apr 17, 2023 10:40 am
by Fred
As long it works, it's not a bug. You don't know how is compiled the 7z.exe file, it might use SSE3 etc to improve decoding speed. If you need a faster decoder, you can use the 7z.exe with RunProgram()
Re: lzma unpack speed, extremely slow on big Archiv
Posted: Tue Apr 18, 2023 5:35 am
by useful
Fred wrote: Mon Apr 17, 2023 10:40 am
As long it works, it's not a bug. You don't know how is compiled the 7z.exe file, it might use SSE3 etc to improve decoding speed. If you need a faster decoder, you can use the 7z.exe with RunProgram()
Will my question be answered in the same way?
viewtopic.php?t=81370
Or can I hope for an update?
Re: lzma unpack speed, extremely slow on big Archiv
Posted: Tue Apr 18, 2023 7:02 am
by BarryG
@Taz, this is the result I get:
Code: Select all
FileSize: 538 MiB, unpack time:
PB: 54917 ms
7zr: 11405 ms
PureBasic is 382% slower
Re: lzma unpack speed, extremely slow on big Archiv
Posted: Tue Apr 18, 2023 7:10 am
by jacdelad
Are there differences between ASM and C? Also, tested with debugger on or with compiled exe (since there are debug commands in the code, I assume you did it in the debugger)?
Re: lzma unpack speed, extremely slow on big Archiv
Posted: Tue Apr 18, 2023 7:11 am
by Fred
useful wrote: Tue Apr 18, 2023 5:35 am
Fred wrote: Mon Apr 17, 2023 10:40 am
As long it works, it's not a bug. You don't know how is compiled the 7z.exe file, it might use SSE3 etc to improve decoding speed. If you need a faster decoder, you can use the 7z.exe with RunProgram()
Will my question be answered in the same way?
viewtopic.php?t=81370
Or can I hope for an update?
Is it the same issue ? So why would it be answered the same way ? Please don't be rude here, thank you.
Re: lzma unpack speed, extremely slow on big Archiv
Posted: Tue Apr 18, 2023 7:17 am
by Bisonte
Code: Select all
FileSize: 538 MiB, unpack time:
PB: 16464 ms
7zr: 4331 ms
PureBasic is 280% slower
But how can it be, that it is faster with debugger. (Not much.... 16800-16900ms without debugger) ??
Re: lzma unpack speed, extremely slow on big Archiv
Posted: Tue Apr 18, 2023 9:39 am
by Taz
jacdelad wrote: Tue Apr 18, 2023 7:10 am
Are there differences between ASM and C? Also, tested with debugger on or with compiled exe (since there are debug commands in the code, I assume you did it in the debugger)?
I've tested it with and without a debugger, and as a compiled exe.
Also with different Purebasic versions (5.5x, 5.6x, 5.7x, 6.01)
Also with c-backend (with and without optimizer)
And I also tried the "Cpu Optimization" (SSE, SSE2) options
Unfortunately no significant differences.
@Fred
I don't know how old the internal lzma code is, maybe updating it would help?
7z Unpack speed has improved a lot in the last few months/years.
(Just a thought from me, who has no idea about asm and c.)
Re: lzma unpack speed, extremely slow on big Archiv
Posted: Tue Apr 18, 2023 10:22 am
by PureGuy
This is all about solid compression of *.7z files.
The Packer library of PureBasic does extract file by file.
If you extract the entire solid 7z archive, all necessary blocks are decompressed again and again.
Re: lzma unpack speed, extremely slow on big Archiv
Posted: Tue Apr 18, 2023 11:14 am
by Taz
PureGuy wrote: Tue Apr 18, 2023 10:22 am
This is all about solid compression of *.7z files.
The Packer library of PureBasic does extract file by file.
If you extract the entire solid 7z archive, all necessary blocks are decompressed again and again.
Sure you're right, but in this example there is only ONE big file in the archive.
Re: lzma unpack speed, extremely slow on big Archiv
Posted: Wed Apr 19, 2023 7:25 pm
by tester
Code: Select all
12th Gen Intel(R) Core(TM) i5-12500
FileSize: 538 MiB, unpack time:
PB: 23554 ms
7zr: 19141 ms
PureBasic is 23% slower
Not such a big difference.