Packer should return progress

[Jens-Arne]

Hi all,

I'd appreciate it if the packer functions (e.g. AddPackFile) would take another parameter with the address of a quadint for 64bit systems and a longint for 32bit systems into which it writes the number of bytes it has already processed.

I know that this is normally useless because the function doesn't return until it has finished its work so you never have a chance to read the progress variable during the actual compression work. But things start to look different if we used the compression function in a thread. Then we'd be able to access the progress variable in the main program during compression and show a progress bar or whatever.

The reason why the parameter should be a quadint for 64bit systems and a longint for 32bit systems is that writing into and reading from the variable must be atomic in order to get correct results without the need to use critical sections. The latter consume an enormous amount of time just for synchronisation between the threads and would thus badly compromise the compression speed. So things have to be atomic to go smoothly. I assume that PureBasic uses atomic machine code to access 64bit integer variables with 64bit operationg systems. If not we would simply need to use longints for both 32bit and 64bit systems.

EDIT: I know that this is something for advanced programmers. But in my opinion it is always nice to be able to show the progress of a potentially very lengthy operation. Compressing large files can take an awful lot of seconds if not minutes.

EDIT 2: The discussion has - for good reason - shifted to a callback function instead of a direct variable address - see posts below.

Best regards, Jens-Arne

[Quin]

+1 on the idea, I'm not sure if I'm crazy about the implemntation though.

[Thorium]

The main reason i don't use the packer lib at all anymore.
This is also possible without a separate thread via callback.

[acreis]

Hi Thorium, what is the alternative to packer lib?

[jacdelad]

+1 for a callback variant.

[Jens-Arne]

@Thorium: Yes indeed, good idea, a parameter that takes the address of a callback function would be even better (no need for threads). The callback function itself would take a QuadInt as progress indicator and, say, a user defined Integer as is often the case with Windows callback functions.

[Fred]

There was such a function in previous PB version, but it's been removed when the packer lib has been revamped. I will see what can be done to reintroduce it (doc is still online https://www.purebasic.com/documentation ... lback.html)

[Jens-Arne]

@Fred: That sounds great! I hope it isn't too much of an effort to put that feat back in. Normally (but what is "normal" in programming?) that should be fairly straightforward so maybe your dev team has a chance to take a look at that somewhen in the future without having to divert too much time to it.

[NicTheQuick]

There was such a function in previous PB version, but it's been removed when the packer lib has been revamped. I will see what can be done to reintroduce it (doc is still online https://www.purebasic.com/documentation ... lback.html)

lol. I remembered that but as I was reading the thread I was like "Huh? Is it gone?".

[Jens-Arne]

Please allow me one more suggestion:

In the original version the packer lib calculated how often it invoked the callback funktion without the possibility for the programmer to change the outcome.
To relieve the lib from this burden (which requires some calculations inside the packing loop, most probably a modulo function, and should therefore be avoided to maximize packing speed) you could leverage the timer functions of Windows/Linux with their own callback functionality. I don't know, however, if this works for MacOS as well - it seems, though, as if it does indeed when looking at Google.

The idea: Let a timer invoke an "internal" callback function that in itself then calls the user definded callback function with the desired progress values. This way the packer lib doesn't have to bother about invoking the user callback function inside the packing functions at all.
The timer value can be chosen by the programmer so one can trade progress resolution against packing speed oneself.

Windows: SetTimer with the fourth parameter set to the "internal" callback function
Linux: timer_create with sevp.sigev_notify set to SIGEV_THREAD and sigev_notify_function to the "internal" callback function

Windows example:

Code: Select all

Procedure InternalPackerCallbackProc(<OS-specific Parameters>)
  Flag_ContinuePacking=CallFunctionFast(<AddressOfUserCallbackProc>,SourcePosition,DestinationPosition)
EndProcedure

SetTimer_(<HandleOfMainWindow>,<IdOfTimer>,<UserChosenTimerValue>,@InternalPackerCallbackProc())
;... do some packing (can be interrupted by setting Flag_ContinuePacking to 0)
KillTimer_(<HandleOfMainWindow>,<IdOfTimer>)

[Thorium]

Hi Thorium, what is the alternative to packer lib?

Using the compressions libs directly like XZ Utils for LZMA.

[DeanH]

I thought this was not possible so had been removed. If it can be done, please add my +100 vote. I found reference to the old PackerCallback in version 4.61. It was discontinued in PB 5. The lack of this feature is the main reason I use 7-zip and LHA dll's instead. They show a progressbar. Lets the user know something is happening when a large file is compressed or extracted.

[jacdelad]

Does anyone want to share the corde for the use of lha, 7zip, xz, the lzma library, etc. (the ones mentioned here), or are they already posted?

[DeanH]

I can't remember / locate where I obtained the 7-zip dll files. The thread here called "Using 7Zip.dll in UNICODE [Resolved]" helped. A date of 2006 appears in a txt file along with the name Akita Minoru. I have both 32-bit and 64-bit versions. A Google search turned up many download sources. A normal 7-zip CLI command is sent to the function "SevenZip". The dll handles zip and 7z / lzma formats. My simple procedure to use them is:

Code: Select all

Procedure$ SevenZip(Cmd$)
	Protected Id,wn,Output$,Integer.i,Zipdll$,OutputAnsi$,*Cmd
	Zipdll$="7-zip32.dll"
	If Bool(#PB_Compiler_Processor = #PB_Processor_x64)
		Zipdll$="7-zip64.dll"
	EndIf
	Id=OpenLibrary(#PB_Any,Zipdll$)
	If Id
		wn=OpenWindow(#PB_Any, 0, 0, 320, 200, "Zip command",#PB_Window_ScreenCentered|#PB_Window_Invisible)
		If wn
			OutputAnsi$=Space(32767)
			*Cmd=Ascii(Cmd$)
			CallFunction(Id,"SevenZip",WindowID(wn),*Cmd,@OutputAnsi$,32767)
			Output$=PeekS(@OutputAnsi$,32767,#PB_Ascii)
			FreeMemory(*Cmd)
			CloseWindow(wn)
		EndIf
		CloseLibrary(Id)
	EndIf
	ProcedureReturn Output$
EndProcedure

I prefer LHA to 7-zip. The range of commands and switches is larger. Unfortunately, I could not find a 64-bit version that both compresses and decompresses. Unlha32.dll shows a dialog with progressbars. Only a 32-bit version dating from 1996 seems to be available. To run from 64-bit, the 32-bit Lha32.exe loader is called. The procedure returns the text generated.

Code: Select all

Procedure$ LHA(wn.i,Cmd$,sc.i=1000000)
	Protected prog, Output$, *mem, whandle, *buffer
	If Cmd$
		If Bool(#PB_Compiler_Processor = #PB_Processor_x64)=1
			prog=RunProgram("Lha32.exe",Cmd$,"",#PB_Program_Open | #PB_Program_Read | #PB_Program_Ascii)
			If prog
				While ProgramRunning(prog)
					If AvailableProgramOutput(prog)
						Output$+ReadProgramString(prog)+#CRLF$
					EndIf
	 			Wend
				CloseProgram(prog)
			EndIf
			ProcedureReturn Output$
		EndIf
	Else
		*mem=AllocateMemory(1024)
		PokeS(*mem,Cmd$,Len(Cmd$),#PB_Ascii)
		If wn>=0
			If IsWindow(wn)
				whandle.i=WindowID(wn)
			EndIf
		Else
			whandle=GetDesktopWindow_()
		EndIf
		Id=OpenLibrary(#PB_Any,"UNLHA32.DLL")
		If Id
			Output$=Space(sc)
			*buffer=AllocateMemory(sc)
			if CallFunction(Id,"Unlha",whandle,*mem,*buffer,sc)
				CloseLibrary(Id)
				Output$=PeekS(*buffer,-1,#PB_Ascii)
				Output$=Trim(Output$)
			EndIf
			FreeMemory(*buffer)
		EndIf
		FreeMemory(*mem)
	EndIf
	ProcedureReturn Output$
EndProcedure