Page 1 of 1

MD5-Hash with callback, progress, elapsed/remaining time etc

Posted: Fri Mar 14, 2008 3:15 pm
by AND51
Hello!

based on Kiffi's code example how to use ExamineMD5Fingerprint() I wrote a procedure with offers you the following:
  • progress in percent
  • elapsed time
  • remaining time
  • speed in byte/second (with my attached bytecalc()-procedure, you can automatically transform bytes into KB, MB, GB, ...)
  • possibility to abort the operation, if callback returns a value <>0
Code:

Code: Select all

EnableExplicit
#file="D:\Dateien\CD Image.img"



Procedure.s bytecalc(byte.q, NbDecimals.l=0)
	If byte < 1024
		If byte < 0
			ProcedureReturn "0 Byte"
		EndIf
		ProcedureReturn Str(byte)+" Byte"
	ElseIf byte >= 1<<60
		ProcedureReturn StrD(byte/1<<60, NbDecimals)+" EB"
	ElseIf byte >= 1<<50
		ProcedureReturn StrD(byte/1<<50, NbDecimals)+" PB"
	ElseIf byte >= 1<<40
		ProcedureReturn StrD(byte/1<<40, NbDecimals)+" TB"
	ElseIf byte >= 1<<30
		ProcedureReturn StrD(byte/1<<30, NbDecimals)+" GB"
	ElseIf byte >= 1<<20
		ProcedureReturn StrD(byte/1<<20, NbDecimals)+" MB"
	Else
		ProcedureReturn StrD(byte/1024, NbDecimals)+" KB"
	EndIf
EndProcedure


Procedure myCallback(file.s, progress.l, speed.l, remainingTime.l, elapsedTime.l)
	SetGadgetText(3, "Entire progress: "+Str(progress)+"%")
	SetGadgetText(4, "Speed: "+bytecalc(speed)+"/sec ("+Str(speed)+" Byte/s)")
	SetGadgetText(5, FormatDate("Elapsed: %hhh%iim%sss", elapsedTime)+#CRLF$+FormatDate("Remaining: %hhh%iim%sss", remainingTime))
	SetGadgetState(0, Int(progress))
	While WindowEvent() : Wend
	ProcedureReturn 0 ; Return 0 to continue, 1 to abort
EndProcedure

Procedure.s MD5FileFingerprintCallback(file$, *callback=0, callCallbackEvery=1000, bufferSize=4096)
	; Callback format: myCallback(file.s, progress.l, speed.l, remainingTime.l, elapsedTime.l)
	Protected startTime.l=Date(), file.l=ReadFile(#PB_Any, file$), hash.s
	If file
		Protected *buffer=AllocateMemory(bufferSize)
		If *buffer
			Protected fingerprint=ExamineMD5Fingerprint(#PB_Any)
			If fingerprint
				Protected progress.l, refresh.l, elapsed.l, speed.l, remaining.l
				While Not Eof(file)
					NextFingerprint(fingerprint, *buffer, ReadData(file, *buffer, bufferSize))
					If *callback And ElapsedMilliseconds() > refresh 
						If Lof(file) ; to avoid division by zero if filesize is 0 bytes
							elapsed=Date()-startTime
							progress=Round(Loc(file)/Lof(file)*100, #PB_Round_Nearest)
							If elapsed ; to avoid division by zero
								speed=Round(Loc(file)/elapsed, #PB_Round_Nearest)
								If speed ; prevent division by zero
									remaining=Round((Lof(file)-Loc(file))/speed, #PB_Round_Nearest)
								EndIf
							EndIf
						EndIf
						If CallFunctionFast(*callback, file$, progress, speed, remaining, elapsed)
							Break
						EndIf
						refresh=ElapsedMilliseconds()+callCallbackEvery
					EndIf
				Wend
				hash=FinishFingerprint(fingerprint)
			EndIf
			FreeMemory(*buffer)
		EndIf
		CloseFile(file)
	EndIf
	ProcedureReturn hash
EndProcedure



  If OpenWindow(0, 0, 0, 400, 160, GetFilePart(#file)+" "+bytecalc(FileSize(#file), 1), #PB_Window_SystemMenu | #PB_Window_ScreenCentered) And CreateGadgetList(WindowID(0))
    ProgressBarGadget(0,  10, 30, 250,  30, 0, 100)
    TextGadget       (3,  10, 10, 250,  20, "")
    TextGadget       (4,  10, 70, 250,  20, "")
    TextGadget       (5, 100,110, 200,  50, "")
    Define md5$=MD5FileFingerprintCallback(#file, @myCallback(), 250)
   MessageRequester("MD5FileFingerprintCallback()", "finished! send donations to AND51 :-P"+#CRLF$+md5$)
  EndIf
file$
File which the MD5 hash code is being examined from.

*callback
Address of the procedure/callback to be called. If *callback=0, then the whole callback-function is being disabled. The callback must have this format:
myCallback(file.s, progress.l, speed.l, remainingTime.l, elapsedTime.l)
The callback is able to abort the operation by returning a value <>0.

callCallbackEvery
Interval in milliseconds in which the callback is being called. This is useful, because if a gadget is being updated too fast, it starts flickering.

bufferSize
Size in bytes for the buffer that should be allocated. A larger buffer can improve the performance.

Returning value
The procedure returns the MD5 hash as a string or an empty string, if there was an error (file cannot be opened, memory allocation error, ...).

Posted: Fri Mar 14, 2008 6:54 pm
by fsw
Thanks.

What amazes me is that PureBasic's own command
MD5FileFingerprint(Filename$)
uses more than double the time of ExamineMD5Fingerprint to do a MD5 operation on the same file. :shock:


BTW: changing the buffer size for NextFingerprint doesn't seem to make a big difference while using ExamineMD5Fingerprint (at least not on this machine)

Posted: Fri Mar 14, 2008 6:57 pm
by AND51
freak said today, that MD5Filefingerprint() uses an internal buffer of about 1 MB. I also don't know, why this command is so slow... :?:
Thx for your feedback.

Posted: Fri Mar 14, 2008 9:54 pm
by fsw
AND51 wrote:freak said today, that MD5Filefingerprint() uses an internal buffer of about 1 MB. I also don't know, why this command is so slow... :?:
Thx for your feedback.
Tested Kiffi's code (test file size 650mb) with several buffer sizes starting from 4kb, 64kb up to 128mb.
The time needed was always around 15 seconds with a difference of 2 to 4 seconds.

MD5Filefingerprint took at least 32 seconds for the same task.

Maybe the PB team should consider to use the ExamineMD5Fingerprint technique inside the MD5Filefingerprint function :P

Posted: Mon Mar 24, 2008 6:24 pm
by moogle
this code doesn't work for me :(
It says
[COMPILER] Line 43: ExamineMD5Fingerprint() is not a function, array, macro or linked list

Using PB 4.1 Windows

Am I doing something wrong?

Posted: Mon Mar 24, 2008 7:14 pm
by walker
... yes.. using the wrong PB-Version... that command is 4.20 only... :wink:

Posted: Mon Mar 24, 2008 9:59 pm
by moogle
walker wrote:... yes.. using the wrong PB-Version... that command is 4.20 only... :wink:
thanks for that. I guess I'll try when 4.2 is released stable :D

Posted: Wed Jul 09, 2008 12:32 pm
by whertz
Is there a way to do the same thing but for CRC32? When using CRC32FileFingerPrint on a big file you just have to wait for it to finish, and there is no way of knowing when it's going to be done, or no way of cancelling it.

Posted: Wed Jul 09, 2008 7:04 pm
by AND51
Yes, you could use CRC32Fingerprint(), which allows you to determine the CRC32 hash in several steps.
Since there is no 'ExamineCRC32Fingerprint()', you must redesign my procedure.

Re:

Posted: Mon Aug 15, 2011 1:01 pm
by moogle
AND51 wrote:Yes, you could use CRC32Fingerprint(), which allows you to determine the CRC32 hash in several steps.
Since there is no 'ExamineCRC32Fingerprint()', you must redesign my procedure.
CRC32 one redesigned, for practice and possibly learners. Many thanks AND51.

Code: Select all

EnableExplicit
#file="E:\Files\Games\Thief\Deadly Shadows\Patches\Collective Texture Pack ver. 1.0.3.exe" ;CRC32: 7C538797


Procedure.s bytecalc(byte.q, NbDecimals.l=0)
   If byte < 1024
      If byte < 0
         ProcedureReturn "0 Byte"
      EndIf
      ProcedureReturn Str(byte)+" Byte"
   ElseIf byte >= 1<<60
      ProcedureReturn StrD(byte/1<<60, NbDecimals)+" EB"
   ElseIf byte >= 1<<50
      ProcedureReturn StrD(byte/1<<50, NbDecimals)+" PB"
   ElseIf byte >= 1<<40
      ProcedureReturn StrD(byte/1<<40, NbDecimals)+" TB"
   ElseIf byte >= 1<<30
      ProcedureReturn StrD(byte/1<<30, NbDecimals)+" GB"
   ElseIf byte >= 1<<20
      ProcedureReturn StrD(byte/1<<20, NbDecimals)+" MB"
   Else
      ProcedureReturn StrD(byte/1024, NbDecimals)+" KB"
   EndIf
EndProcedure



Procedure myCallback(file.s, progress.l, speed.l, remainingTime.l, elapsedTime.l)
   SetGadgetText(3, "Entire progress: "+Str(progress)+"%")
   SetGadgetText(4, "Speed: "+bytecalc(speed)+"/sec ("+Str(speed)+" Byte/s)")
   SetGadgetText(5, FormatDate("Elapsed: %hhh%iim%sss", elapsedTime)+#CRLF$+FormatDate("Remaining: %hhh%iim%sss", remainingTime))
   SetGadgetState(0, Int(progress))
   While WindowEvent() : Wend
   ProcedureReturn 0 ; Return 0 to continue, 1 to abort
EndProcedure

Procedure.s CRC32FileFingerprintCallback(file$, *callback=0, callCallbackEvery=1000, bufferSize=4096)
   ; Callback format: myCallback(file.s, progress.l, speed.l, remainingTime.l, elapsedTime.l)
   Protected startTime.l=Date(), file.l=ReadFile(#PB_Any, file$), hash.s
   If file
      Protected *buffer=AllocateMemory(bufferSize)
      If *buffer
			  Protected progress.l, refresh.l, elapsed.l, speed.l, remaining.l, CRC32.l
			  While Not Eof(file)
			  		CRC32=CRC32Fingerprint(*buffer, ReadData(file, *buffer, bufferSize), CRC32)
			     If *callback And ElapsedMilliseconds() > refresh
			        If Lof(file) ; to avoid division by zero if filesize is 0 bytes
			           elapsed=Date()-startTime
			           progress=Round(Loc(file)/Lof(file)*100, #PB_Round_Nearest)
			           If elapsed ; to avoid division by zero
			              speed=Round(Loc(file)/elapsed, #PB_Round_Nearest)
			              If speed ; prevent division by zero
			                 remaining=Round((Lof(file)-Loc(file))/speed, #PB_Round_Nearest)
			              EndIf
			           EndIf
			        EndIf
			        If CallFunctionFast(*callback, @file$, progress, speed, remaining, elapsed)
			           Break
			        EndIf
			        refresh=ElapsedMilliseconds()+callCallbackEvery
			     EndIf
			  Wend
			  hash=Hex(CRC32, #PB_Long)
      	FreeMemory(*buffer)
      EndIf
      CloseFile(file)
   EndIf
   ProcedureReturn hash
EndProcedure


  If OpenWindow(0, 0, 0, 400, 160, GetFilePart(#file)+" "+bytecalc(FileSize(#file), 1), #PB_Window_SystemMenu | #PB_Window_ScreenCentered)
    ProgressBarGadget(0,  10, 30, 250,  30, 0, 100)
    TextGadget       (3,  10, 10, 250,  20, "")
    TextGadget       (4,  10, 70, 250,  20, "")
    TextGadget       (5, 100,110, 200,  50, "")
    Define c0=timeGetTime_()
    Define crc32$=CRC32FileFingerprintCallback(#file, @myCallback(), 50, 256*1024)
  	MessageRequester("Finished", "CRC: "+crc32$+#CRLF$+"Time: "+Str(timeGetTime_()-c0)+"ms")
  EndIf