Page 1 of 2

PeekHexBytes() and PokeHexBytes()

Posted: Wed Aug 18, 2010 2:17 pm
by Little John
Works also with PB 5.20

Hi all,

here are two basic cross-platform functions, that are useful for reading arbitrary content from memory, to store it into strings or text files, and to write it back to memory. PeekHexBytes() comes in handy for instance in context with PB's AESEncoder() (e.g. in order to store encrypted passwords in a preference file), since the encoded output may contain bytes with value 0, so that PB's PeekS() cannot be used safely for reading it. PokeHexBytes() is the complementary function, which allows to easily write arbitrary byte sequences to memory.
( Another option for getting only byte values that can be stored in strings, is e.g. conversion with PB's Base64Encoder(). )

I hope the code is useful for some of you. It is designed for flexibility, simplicity and safety. It's not optimized for speed, since that can easily result in bad readable code, and would mean premature optimization in probably most cases. So do speed optimization yourself if you need it, and only when you need it.

Regards, Little John

//edit 2010-08-24
PokeHexBytes() completely rewritten:
Thanks to Trond, the procedure now accepts hexadecimal strings in more different input formats, and is much simpler at the same time. On error, it now returns -1 instead of 0.

Code: Select all

; -- Tested with PB 4.51 RC 2 x86 on Windows XP and Ubuntu 10.04,
;    compiled to Unicode-executable and to non-Unicode-executable

EnableExplicit

Procedure.s PeekHexBytes (*source.Ascii, numBytes=-1, separator$="", prefix$="")
   ; -- returns a hexadecimal representation of the bytes in a memory area
   ;    ("Hex dump")
   ; in : *source   : points to begin of the source memory area
   ;      numBytes  : number of bytes to read (< 0 reads up to next null byte)
   ;      separator$: string that is written as separator in the output
   ;      prefix$   : prefix for each hex byte in the output (e.g. "$" or "0x")
   ; out: returns a string of unsigned hexadecimal byte values;
   ;      hex digits "A" to "F" are upper case.
   Protected *eofSource, ret$=""

   *eofSource = *source + numBytes
   While *source < *eofSource Or (*source > *eofSource And *source\a <> 0)
      ret$ + separator$ + prefix$ + RSet(Hex(*source\a), 2, "0")
      *source + 1
   Wend

   ProcedureReturn Mid(ret$, Len(separator$)+1)
EndProcedure


Macro SkipString (_s_, _length_)
   If CompareMemory(*source, @_s_, _length_)
      *source + _length_
   EndIf
EndMacro

Procedure.i IsHexDigit (*source.Character)
   Select *source\c
      Case '0' To '9', 'A' To 'F', 'a' To 'f'
         ProcedureReturn #True
      Default      
         ProcedureReturn #False
   EndSelect
EndProcedure

Procedure.i PokeHexBytes (*dest.Ascii, hexList$, separator$="", prefix$="")
   ; -- writes bytes, that are given in the form of a string of hex
   ;    values, to a memory area
   ; in : *dest     : points to begin of the destination memory area
   ;      hexList$  : list of unsigned hexadezimal byte values;
   ;                  Each byte must be represented by 2 hex digits, and
   ;                  can optionally have a prefix (such as "$" or "0x").
   ;                  Hex digits "A" to "F" can be upper or lower case.
   ;      separator$: string that is used as separator between bytes in
   ;                  hexList$ (e.g. " ", ";" or "; ")
   ;      prefix$   : prefix that is used for bytes in hexList$
   ; out: returns number of written bytes on success, or -1 on error
   Protected *destStart, *source.Character, lenPre, lenSep

   *destStart = *dest
   *source = @hexList$
   lenPre = StringByteLength(prefix$)
   lenSep = StringByteLength(separator$)
   
   While *source\c
      SkipString(prefix$, lenPre)
      If (Not IsHexDigit(*source)) Or (Not IsHexDigit(*source+SizeOf(Character)))
         ProcedureReturn -1      ; error
      EndIf   
      *dest\a = Val("$" + PeekS(*source, 2))
      *source + 2*SizeOf(Character)
      *dest + 1
      SkipString(separator$, lenSep)
   Wend
   
   ProcedureReturn *dest - *destStart
EndProcedure


;=======================================================================
;                  B E G I N   O F   D E M O   C O D E
;=======================================================================


Define numBytes, i, *source, *destination, hexList$

numBytes = 20

*source      = AllocateMemory(100)
*destination = AllocateMemory(100)


; --- Fill the source buffer with some values, and with trailing null ---
For i = 0 To numBytes - 1
   PokeA(*source+i, i+1)
Next
PokeA(*source+numBytes, 0)


; --- Get hexadecimal representation of the source, write the bytes to
;     target, and compare source with target  ==>  should result in 1  ---

hexList$ = PeekHexBytes(*source)
Debug hexList$
Debug PokeHexBytes(*destination, hexList$)
Debug CompareMemory(*source, *destination, numBytes)
PokeA(*destination, 0)            ; partially overwrite destination

hexList$ = PeekHexBytes(*source, numBytes, " ")
Debug hexList$
Debug PokeHexBytes(*destination, hexList$, " ")
Debug CompareMemory(*source, *destination, numBytes)
PokeA(*destination, 0)            ; partially overwrite destination

hexList$ = PeekHexBytes(*source, -1, ",")
Debug hexList$
Debug PokeHexBytes(*destination, hexList$, ",")
Debug CompareMemory(*source, *destination, numBytes)
PokeA(*destination, 0)            ; partially overwrite destination

hexList$ = PeekHexBytes(*source, -1, "", "$")
Debug hexList$
Debug PokeHexBytes(*destination, hexList$, "", "$")
Debug CompareMemory(*source, *destination, numBytes)
PokeA(*destination, 0)            ; partially overwrite destination

hexList$ = PeekHexBytes(*source, numBytes, " ", "$")
Debug hexList$
Debug PokeHexBytes(*destination, hexList$, " ", "$")
Debug CompareMemory(*source, *destination, numBytes)
PokeA(*destination, 0)            ; partially overwrite destination

hexList$ = PeekHexBytes(*source, -1, ",", "$")
Debug hexList$
Debug PokeHexBytes(*destination, hexList$, ",", "$")
Debug CompareMemory(*source, *destination, numBytes)
PokeA(*destination, 0)            ; partially overwrite destination

hexList$ = PeekHexBytes(*source, -1, "", "0x")
Debug hexList$
Debug PokeHexBytes(*destination, hexList$,  "", "0x")
Debug CompareMemory(*source, *destination, numBytes)
PokeA(*destination, 0)            ; partially overwrite destination

hexList$ = PeekHexBytes(*source, numBytes, " ", "0x")
Debug hexList$
Debug PokeHexBytes(*destination, hexList$, " ", "0x")
Debug CompareMemory(*source, *destination, numBytes)
PokeA(*destination, 0)            ; partially overwrite destination

hexList$ = PeekHexBytes(*source, -1, ", ", "0x")
Debug hexList$
Debug PokeHexBytes(*destination, hexList$, ", ", "0x")
Debug CompareMemory(*source, *destination, numBytes)
PokeA(*destination, 0)            ; partially overwrite destination

Debug "----------------"


; --- Some hexadecimal strings that are NOT accepted by PokeHexBytes() ---

Debug PokeHexBytes(*destination, "123")               ; odd number of hex digitis

Debug PokeHexBytes(*destination, "12 3 45", " ")      ; only one digit between separators

Debug PokeHexBytes(*destination, "12FX")              ; invalid hex digit "X"

Debug PokeHexBytes(*destination, "12,34")             ; invalid separator

Debug PokeHexBytes(*destination, ",12", ",")          ; separator at the beginning

Debug PokeHexBytes(*destination, "12,,34", ",")       ; separator followed by separator

Debug PokeHexBytes(*destination, "12$,34", ",", "$")  ; prefix followed by separator


FreeMemory(*source)
FreeMemory(*destination)

Re: PeekHexBytes() and PokeHexBytes()

Posted: Wed Aug 18, 2010 5:29 pm
by Rescator
I couldn't resist improving this one.

First of all I removed the unknown length "feature", far too easy to get bugs if one is careless, fixed lengths are always best when dealing with binary.
I also added another prefix feature, this one is only shown at the start of the string and only if no separator is used.

I also tried to make PeekHexBytes() as efficient as I could, I'm sure there are still ways to speed it up some more, maybe some of the other oldtimers here got a few tips on that!

I also made PokeHexBytes() extremely flexible, just look at test 5, hopefully some form of input sanitation is done to prevent crap like that, but test 5 is just to show how simple yet powerful the parsing is. The only flaws of the parser is that it can't notice broken hex pairs (ie if it was "01 0 03" it would assume this is "01 00" and it would ignore the 3 as it would assume the 3 was a lost digit instead.) Then again that's a nice tradeoff for such flexible prefix, separation support.
Also PokeHexByte() will tell you how many digits it wrote.So in test 5 it returns 19 instead of the expected 20 so you can detect bad hex string lengths that way.

It should work equally well on Ascii and Unicode (unless I messed something up) and any x86 platform (not sure about the nibbles and bit shifting on PowerPC)
Oh and the new source/changes that I added is obviously public domain, have fun with it.

Code: Select all

EnableExplicit

Structure _chars_struct_
	StructureUnion
		s.s{2}
		c.c[2]
	EndStructureUnion
EndStructure

Procedure.s PeekHexBytes (*source.Byte, numBytes.i, separator$="", prefix$="")
   ; -- returns the hexadecimal values of the bytes in a memory area
   ;    ("HexDump")
   ; in : *source   : points to begin of the source memory area
   ;      numBytes  : number of bytes to be read (< 0 reads up to next 0 byte)
   ;      separator$: string that is used as separator in the output
   ;      prefix$   : prefix that is used for each hex value in the output;
   ; out: string of unsigned hexadecimal values of the bytes from the respective
   ;      memory area
   Protected *eofSource, ret$, byte1.i, byte2.i, chars._chars_struct_
   
   *eofSource=*source+numBytes
	While (*source<*eofSource)
		byte1=((*source\b>>4)&%1111)
		byte2=(*source\b&%1111)
		If (byte1<10)
			chars\c[0]=byte1+48
		Else
			chars\c[0]=byte1+87
		EndIf
		If (byte2<10)
			chars\c[1]=byte2+48
		Else
			chars\c[1]=byte2+87
		EndIf
		If prefix$ And separator$=""
			ret$+chars\s
		Else
			If (*source<(*eofSource-1))
				ret$+prefix$+chars\s+separator$
			Else
				ret$+prefix$+chars\s
			EndIf
		EndIf
		*source+1
	Wend
   
	If prefix$ And separator$=""
	   ProcedureReturn prefix$+ret$
	Else
	   ProcedureReturn ret$
	EndIf
EndProcedure

Procedure.i PokeHexBytes (*dest.Byte, numBytes.i, byteList$)
   ; -- writes bytes that are given in the form of a string of hex
   ;    values, to a memory area
   ; in: *dest    : points to begin of the destination memory area
   ;     numBytes: size of the destination memory
   ;     byteList$: list of unsigned hexadecimal byte values
   ; out: len		:the number of numbers parsed
   Protected *source.Character, *eofSource, byte1.i, byte2.i, *eofDest, len.i
   *source = @byteList$
   *eofSource = *source + StringByteLength(byteList$)
   *eofDest = *dest + numBytes
	While (*source < *eofSource)
		byte1=*source\c
		If (((byte1>47) And (byte1<58)) Or ((byte1>96) And (byte1<103)) Or ((byte1>64) And (byte1<71)))
			Repeat
		      *source + SizeOf(Character)
	      	If (*source < *eofSource)
					byte2=*source\c
					If (((byte2>47) And (byte2<58)) Or ((byte2>96) And (byte2<103)) Or ((byte2>64) And (byte2<71)))
						If *dest < *eofDest
			      		If byte1<65
			      			*dest\b=((byte1-48)<<4)
			      		ElseIf byte1>96
			      			*dest\b=((byte1-87)<<4)
			      		Else
			      			*dest\b=((byte1-55)<<4)
			      		EndIf
			      		If byte2<65
			      			*dest\b+(byte2-48)
			      		ElseIf byte2>96
			      			*dest\b+(byte2-87)
			      		Else
			      			*dest\b+(byte2-55)
			      		EndIf
			      		len+1
						Else
							Break 2
						EndIf
						*dest + 1
						Break 1
					EndIf
				Else
					Break 2
				EndIf
			Forever
		EndIf
      *source + SizeOf(Character)
	Wend
	ProcedureReturn len
EndProcedure

;-----------------------------------------------------------------------

; * Demo *
Define numBytes, i, *source, *destination, hexList$

numBytes = 20

*source = AllocateMemory(numBytes)
*destination = AllocateMemory(numBytes)

; Fill the source buffer with some values
For i = 0 To numBytes-1
   PokeA(*source+i, i)
Next

; Get source in hexadecimal form, then write
; hex values to target
Debug ""
Debug "Test1"
hexList$ = PeekHexBytes(*source,numbytes)
Debug hexList$
PokeHexBytes(*destination, numBytes, hexList$)
hexList$ = PeekHexBytes(*destination,numBytes)
Debug hexList$

; Get source in hexadecimal form, then write
; hex values to target
Debug ""
Debug "Test2"
hexList$ = PeekHexBytes(*source, numBytes, "", "$")
Debug hexList$
PokeHexBytes(*destination, numBytes, hexList$)
hexList$ = PeekHexBytes(*destination,numbytes)
Debug hexList$

; Get source in hexadecimal form, then write
; hex values to target
Debug ""
Debug "Test3"
hexList$ = PeekHexBytes(*source, numBytes, " ")
Debug hexList$
PokeHexBytes(*destination, numBytes, hexList$)
hexList$ = PeekHexBytes(*destination,numBytes)
Debug hexList$

; Get source in hexadecimal form, then write
; hex values to target
Debug ""
Debug "Test4"
hexList$ = PeekHexBytes(*source, numBytes, ",", "$")
Debug hexList$
PokeHexBytes(*destination, numBytes, hexList$)
hexList$ = PeekHexBytes(*destination,numBytes)
Debug hexList$

; Get source in hexadecimal form, then write
; hex values to target
Debug ""
Debug "Test5"
hexList$ = PeekHexBytes(*source, numBytes, ",", "$")
Debug "Clean original: "+hexList$
hexList$="$ 00klm01 $02 03  04 - 05.$06 $07 $08 $09 0a,$0b,$0c $0d:$0E,$0F $10x$11x$12 $9"
Debug "Messed up: "+hexList$
Define len.i
len=PokeHexBytes(*destination, numBytes, hexList$)
hexList$ = PeekHexBytes(*destination,len)
Debug "Salvaged: "+hexList$

FreeMemory(*source)
FreeMemory(*destination)

Re: PeekHexBytes() and PokeHexBytes()

Posted: Wed Aug 18, 2010 6:18 pm
by Little John
Rescator wrote:I couldn't resist improving this one.

First of all I removed the unknown length "feature" [...]
It seems that we have different understandings of the term "improvement". :D

Re: PeekHexBytes() and PokeHexBytes()

Posted: Wed Aug 18, 2010 7:33 pm
by blueznl
My take on these (some old code):

Code: Select all

Procedure x_pokehex(addr.i,s.s)                                      ; write a hex sequence to memory
  Protected a.i, b.i, c.i, p.i, q.i, l.i
  ;
  ; *** write a sequence of hex bytes into memory
  ;
  ; in:       addr.i    - memory address
  ;           s.s       - string to write, for example: "$AA0F123C" etc.
  ; retval:             - none
  ; out:                - none
  ;
  ; note: spaces and $ symbols will be automatically skipped
  ;
  p = @s
  c = 0
  l = Len(s)
  ;
  While c+1 < l
    If PeekC(p) = 32 Or PeekC(p) = '$'
      p = p+#x_charsize
      c = c+1
    Else
      a = PeekC(p)-48
      If a > 10
        a = a-7
      EndIf
      b = PeekC(p+#x_charsize)-48
      If b > 10
        b = b-7
      EndIf
      PokeB(addr+q,a*16+b)
      p = p+2*#x_charsize
      q = q+1
      c = c+2
    EndIf
  Wend
EndProcedure

Procedure.s x_peekhex(*addr.byte,length.i,mode.i=7)                  ; read a sequence of bytes from mem and store them in hex format in a string
  Protected s.s, b.l, n.l
  ;
  ; *** read a sequence of bytes from memory and store them in hex format in a string
  ;
  ; in:      *addr.byte             - address where to peek
  ;          length.i               - number of bytes
  ;          [ mode.i = 0 ]         - just a hex string
  ;                     1           - add spaces between the bytes
  ;                     2           - add string sign in front
  ;                     3           - add both
  ;                     7 (default) - add string, add spaces, and show the char itself
  ; retval:  .s                     - string containing hex presentation of peeked data
  ;
  ;
  If (mode & 2) <> 0
    s = "$"
  EndIf
  ;
  n = 0
  While n < Length
    b = *addr\b
    If (mode & 1) <> 0
      s = s+" "
    EndIf
    s = s+Right("0"+Hex(b),2)
    n = n+1
    *addr = *addr+1
    If (mode & 4) <> 0
      If b > 31 And b < 'z'
        s = s+" [ "+Chr(b)+" ]"
      EndIf
    EndIf
  Wend
  ProcedureReturn Trim(s)
EndProcedure

Re: PeekHexBytes() and PokeHexBytes()

Posted: Wed Aug 18, 2010 7:36 pm
by Trond
Nice idea!
Is there a good reason for not allowing prefixes when the separator is empty?

Re: PeekHexBytes() and PokeHexBytes()

Posted: Wed Aug 18, 2010 7:53 pm
by Joakim Christiansen

Re: PeekHexBytes() and PokeHexBytes()

Posted: Fri Aug 20, 2010 11:15 am
by Little John
blueznl wrote:My take on these (some old code):
Your poking procedure is more flexible than mine, and pretty much straightforward. I'll probably change my code for poking, so that it'll work similar to yours. :-) Thanks!
BTW: At the beginning of your code, something like #x_charsize = SizeOf(Character) is missing.

Trond wrote:Nice idea!
Thanks. :-)
Trond wrote:Is there a good reason for not allowing prefixes when the separator is empty?
This is due to the way I wrote the procedure PokeHexBytes(): When the source doesn't contain a separator, then the procedure assumes that there are no other characters but hexadecimal digits in the source. I just couldn't imagine to have a source string such as

Code: Select all

$AB$1F$30
But IMHO blueznl's code is better, it doesn't have this limitation.

Joakim Christiansen wrote:My commands posted long time ago:
http://www.purebasic.fr/english/viewtop ... 19&start=0
Oh, I didn't know about them. Well, now people have several versions to choose from. :-)

Regards, Little John

Re: PeekHexBytes() and PokeHexBytes()

Posted: Fri Aug 20, 2010 9:51 pm
by Little John
Inspired by blueznl's procedure for poking, I completely rewrote PokeHexBytes(). Thanks!
Please see first post for details about all changes, and for the new code.

Regards, Little John

Re: PeekHexBytes() and PokeHexBytes()

Posted: Sat Aug 21, 2010 2:58 pm
by SFSxOI
Here is what I get from your example. Is this correct?

Code: Select all

ECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
20
1
EC ED EE EF F0 F1 F2 F3 F4 F5 F6 F7 F8 F9 FA FB FC FD FE FF
20
1
EC,ED,EE,EF,F0,F1,F2,F3,F4,F5,F6,F7,F8,F9,FA,FB,FC,FD,FE,FF
20
1
$EC$ED$EE$EF$F0$F1$F2$F3$F4$F5$F6$F7$F8$F9$FA$FB$FC$FD$FE$FF
20
1
$EC $ED $EE $EF $F0 $F1 $F2 $F3 $F4 $F5 $F6 $F7 $F8 $F9 $FA $FB $FC $FD $FE $FF
20
1
$EC,$ED,$EE,$EF,$F0,$F1,$F2,$F3,$F4,$F5,$F6,$F7,$F8,$F9,$FA,$FB,$FC,$FD,$FE,$FF
20
1
----------------
0
0
0
0
0
0
0
0
0

Re: PeekHexBytes() and PokeHexBytes()

Posted: Sat Aug 21, 2010 3:54 pm
by Little John
Hi SFSxOI,

yes, that's exactly the correct output.

//edit: That was the correct output of an obsolete version.

Regards, Little John

Re: PeekHexBytes() and PokeHexBytes()

Posted: Sat Aug 21, 2010 4:12 pm
by SFSxOI
Oki dokee then, works with PB 4.50 and Windows 7 Ultimate x86

Thanks :)

Re: PeekHexBytes() and PokeHexBytes()

Posted: Sun Aug 22, 2010 8:12 pm
by Trond
I don't see the point of making PokeHexBytes() so complicated. Here's a variant of just 17 lines (including a helper procedure) that accepts any multi-character separator and any multi-character prefix, at the same time or with the other as an empty string. (The original PeekHexBytes() is included for making test input data and verifying output data.)

Code: Select all

Procedure.s PeekHexBytes (*source.Byte, numBytes=-1, separator$="", prefix$="")
   ; -- returns the hexadecimal representation of the bytes in a memory area
   ;    ("HexDump")
   ; in : *source   : points to begin of the source memory area
   ;      numBytes  : number of bytes to read (< 0 reads up to next null byte)
   ;      separator$: string that is used as separator in the output
   ;      prefix$   : prefix that is used for each hex byte in the output
   ; out: returns a string of unsigned hexadecimal values of the bytes in the
   ;      given memory area; hex digits "A" to "F" are upper case.
   Protected *eofSource, ret$=""
   
   *eofSource = *source + numBytes
   While *source < *eofSource Or (*source > *eofSource And *source\b <> 0)
      ret$ + separator$ + prefix$ + RSet(Hex(*source\b,#PB_Byte), 2, "0")
      *source + 1
   Wend
   
   ProcedureReturn Mid(ret$, Len(separator$)+1)
EndProcedure

; Shorter PokeHexBytes() (warning: input must be correct):
Procedure _GetString(*M.Character, S.s)
  l = StringByteLength(S)
  If CompareMemory(*M, @S, L)
    ProcedureReturn *M+l
  EndIf
  ProcedureReturn *M
EndProcedure

Procedure.s PokeHexBytes(*Dest.Ascii, Bytelist.s, separator.s = "", prefix.s = "")
  Protected *Source.Character = @Bytelist
  While *Source\c
    *Source = _GetString(*Source, Prefix)
    PokeA(*Dest, Val("$" + PeekS(*Source, 2)))
    *Source + 2*SizeOf(Character)
    *Dest + 1
    *Source = _GetString(*Source, separator)
  Wend
EndProcedure

*dest = AllocateMemory(100)

T.s = "Hello World!"

sep.s = ", "
pf.s  = "0xh"

S.s = PeekHexBytes(@T, -1, sep, pf)
Debug S
PokeHexBytes(*dest, s, sep, pf)
Debug PeekHexBytes(*dest, -1, sep, pf)

Re: PeekHexBytes() and PokeHexBytes()

Posted: Sun Aug 22, 2010 10:43 pm
by Little John
@SFSxOI:
You are welcome. :-)


@Trond:

I like your code for poking. It is simpler than mine and even more flexible, since it can handle prefixes such as "0xh" (while I was only thinking of PB's hex prefix).
Anyway ...
I don't see the point of making PokeHexBytes() so complicated.

Code: Select all

; Shorter PokeHexBytes() (warning: input must be correct):
My PokeHexBytes() became so complicated, because I wanted it to recognize incorrect input for safety's sake. However, if for instance PokeHexBytes() only gets as input what PeekHexBytes() has returned as output, then such strict syntax checking is of course overkill.

Regards, Little John

Re: PeekHexBytes() and PokeHexBytes()

Posted: Mon Aug 23, 2010 10:20 am
by Little John
Hi Trond,

I've added some syntax checking to your code, trying to keep things as simple as possible.

Code: Select all

*** Obsolete code removed. See first post for current code. ***
What do you think?

Regards, Little John

Re: PeekHexBytes() and PokeHexBytes()

Posted: Mon Aug 23, 2010 1:03 pm
by yrreti
Hi Little John

You forgot to add the Procedure.s PeekHexBytes