lzma unpack speed, extremely slow on big Archiv

[Taz]

When I unpack a large archive with Purebasic, it takes a very long time. (There is only one file in the archive)
Here is a comparison with the console version of 7zr.exe

Code: Select all

FileSize: 538 MiB, unpack time:
  PB: 18230 ms
 7zr:  4215 ms
PureBasic is 333% slower

FileSize: 1076 MiB, unpack time:
  PB: 31396 ms
 7zr:  4921 ms
PureBasic is 538% slower

FileSize: 2128 MiB, unpack time:
  PB: 67244 ms
 7zr:  6961 ms
PureBasic is 866% slower

7zr.exe needs only 7 seconds for a packed 4GB file, Purebasic needs 67 seconds

Here is an example code:
You can download a 1GB packed (538MB) test file HERE. (the packed file contains only hex characters)
The required 7zr.exe HERE, or directly from the manufacturer.

Code: Select all

EnableExplicit
UseLZMAPacker()

Procedure.q _unpackPB(Archiv.s)
	Protected File.s, pt.q = ElapsedMilliseconds()

	If OpenPack(0, Archiv, #PB_PackerPlugin_Lzma) And ExaminePack(0)
		While NextPackEntry(0)
			If PackEntryType(0) = #PB_Packer_File
				File = GetTemporaryDirectory() + PackEntryName(0)
				UncompressPackFile(0, File)
				Break
			EndIf
		Wend
		ClosePack(0)
	EndIf
	pt = ElapsedMilliseconds() - pt

	If FileSize(File)
		DeleteFile(File)
	EndIf

	ProcedureReturn pt
EndProcedure

Procedure.q _unpack7z(Archiv.s)
	Protected Dir.s = GetTemporaryDirectory() + "~bin", pt.q = ElapsedMilliseconds()

	RunProgram("7zr.exe", ReplaceString("x -aoa '"+Archiv+"' -o'"+Dir+"'", "'", #DQUOTE$), "", #PB_Program_Hide|#PB_Program_Wait)
	pt = ElapsedMilliseconds() - pt

	If FileSize(Dir) = -2
		DeleteDirectory(Dir, "")
	EndIf

	 ProcedureReturn pt
EndProcedure

Structure tTEST
	File.s
	PureBasic.q
	SevenZip.q
EndStructure

If FileSize("7zr.exe") = -1
	MessageRequester("", "pls, download 7zr.exe first!")
	End
EndIf

Define Str.s, Diff.d, NewList Test.tTEST()
AddElement(Test()): Test()\File = "bin.7z"

ForEach Test(): With Test()
	If FileSize(\File) > 0
		\PureBasic = _unpackPB(\File)
		\SevenZip  = _unpack7z(\File)

		Str = "FileSize: " + Str(FileSize(\File)/1024/1024) + ~" MiB, unpack time:\n" +
		      "  PB: " + RSet(Str(\PureBasic), 5)+ ~" ms\n" +
		      " 7zr: " + RSet(Str(\SevenZip), 5) + ~" ms\n"

		Diff = \PureBasic / \SevenZip * 100 - 100
		If Diff > 0
			Str + "PureBasic is " + StrD(Diff, 0) + ~"% slower\n"
		Else
			Str + "PureBasis is faster!\n"
		EndIf
	Else
		Str = "file '" + \File + "' not found!"
	EndIf

	CompilerIf #PB_Compiler_Debugger
		Debug Str
	CompilerElse
		MessageRequester("Unpack speed", Str)
	CompilerEndIf
EndWith:Next

[Fred]

As long it works, it's not a bug. You don't know how is compiled the 7z.exe file, it might use SSE3 etc to improve decoding speed. If you need a faster decoder, you can use the 7z.exe with RunProgram()

[useful]

As long it works, it's not a bug. You don't know how is compiled the 7z.exe file, it might use SSE3 etc to improve decoding speed. If you need a faster decoder, you can use the 7z.exe with RunProgram()

Will my question be answered in the same way?
viewtopic.php?t=81370
Or can I hope for an update?

[BarryG]

@Taz, this is the result I get:

Code: Select all

FileSize: 538 MiB, unpack time:
  PB: 54917 ms
 7zr: 11405 ms
PureBasic is 382% slower

[jacdelad]

Are there differences between ASM and C? Also, tested with debugger on or with compiled exe (since there are debug commands in the code, I assume you did it in the debugger)?

[Fred]

As long it works, it's not a bug. You don't know how is compiled the 7z.exe file, it might use SSE3 etc to improve decoding speed. If you need a faster decoder, you can use the 7z.exe with RunProgram()

Will my question be answered in the same way?
viewtopic.php?t=81370
Or can I hope for an update?

Is it the same issue ? So why would it be answered the same way ? Please don't be rude here, thank you.

[Bisonte]

Code: Select all

FileSize: 538 MiB, unpack time:
  PB: 16464 ms
 7zr:  4331 ms
PureBasic is 280% slower

But how can it be, that it is faster with debugger. (Not much.... 16800-16900ms without debugger) ??

[Taz]

Are there differences between ASM and C? Also, tested with debugger on or with compiled exe (since there are debug commands in the code, I assume you did it in the debugger)?

I've tested it with and without a debugger, and as a compiled exe.
Also with different Purebasic versions (5.5x, 5.6x, 5.7x, 6.01)
Also with c-backend (with and without optimizer)
And I also tried the "Cpu Optimization" (SSE, SSE2) options
Unfortunately no significant differences.

@Fred
I don't know how old the internal lzma code is, maybe updating it would help?
7z Unpack speed has improved a lot in the last few months/years.
(Just a thought from me, who has no idea about asm and c.)

[PureGuy]

This is all about solid compression of *.7z files.
The Packer library of PureBasic does extract file by file.
If you extract the entire solid 7z archive, all necessary blocks are decompressed again and again.

[Taz]

This is all about solid compression of *.7z files.
The Packer library of PureBasic does extract file by file.
If you extract the entire solid 7z archive, all necessary blocks are decompressed again and again.

Sure you're right, but in this example there is only ONE big file in the archive.

[tester]

Code: Select all

12th Gen Intel(R) Core(TM) i5-12500
FileSize: 538 MiB, unpack time:
  PB: 23554 ms
 7zr: 19141 ms
PureBasic is 23% slower

Not such a big difference.