It is currently Thu Dec 03, 2020 4:48 am

All times are UTC + 1 hour




Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: HEX string generating - extreme fast
PostPosted: Mon Apr 18, 2016 9:28 pm 
Offline
Addict
Addict

Joined: Sat Mar 02, 2013 9:17 am
Posts: 930
For the latest string encryption inside QAES the PB Hex generating was absolutely to slow.

I have maked this little routine for very quick generating big HEX strings.

Test it, it is very quick and works with Ascii and Unicode

Werner Albus - http://www.quick-aes-256.de - http://www.nachtoptik.de
Code:
*buffer=AllocateMemory(1e6) ; Buffer to encoding
len_string_bytes=MemorySize(*buffer)
RandomData(*buffer, len_string_bytes)

Repeat
  string_1$+RSet(Hex(i), 2, "0")
  i+1
Until i=256

time=ElapsedMilliseconds()

string$=Space(len_string_bytes*2)
len_string_bytes-1
If SizeOf(character)>1
  Repeat
    PokeL(@string$+iii, PeekL(@string_1$+PeekA(*buffer+iiii)*4))
    iiii+1 : iii+4
  Until iiii>len_string_bytes
Else
  Repeat
    PokeW(@string$+iii, PeekW(@string_1$+PeekA(*buffer+iiii)*2))
    iiii+1 : iii+2
  Until iiii>len_string_bytes
EndIf

Debug (ElapsedMilliseconds()-time)
Debug string$
time=ElapsedMilliseconds()

string$="" : i=0 ; PB variante
Repeat
  string$+RSet(Hex(PeekA(*buffer+i)), 2, "0")
  i+1
Until i=len_string_bytes

Debug (ElapsedMilliseconds()-time)

_________________
http://www.nachtoptik.de


Last edited by walbus on Sat Apr 23, 2016 10:18 pm, edited 2 times in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Mon Apr 18, 2016 9:59 pm 
Offline
Always Here
Always Here

Joined: Sun Sep 07, 2008 12:45 pm
Posts: 5075
Location: Germany
Hi,

if debug is enabled, you get never correct results.
You should use MessageRequester() to output the time.

And << 1 is always faster than * 2
<< 2 is faster than * 4

And your Hex() version is slow because you always add something to a string.
It's not Hex() which is sooo slow.

Better comparrison:
Code:
*buffer=AllocateMemory(1e6) ; Buffer to encoding
len_string_bytes=MemorySize(*buffer)
RandomData(*buffer, len_string_bytes)

Repeat
  string_1$+RSet(Hex(i), 2, "0")
  i+1
Until i=256


time=ElapsedMilliseconds()
string$=Space(len_string_bytes*2)
iiii=0
iii = 0
Repeat
  CompilerIf #PB_Compiler_Unicode
    PokeL(@string$+iii, PeekL(@string_1$+PeekA(*buffer+iiii) << 2))
    iii+4
  CompilerElse
    PokeW(@string$+iii, PeekW(@string_1$+PeekA(*buffer+iiii) << 1))
    iii+2
  CompilerEndIf
  iiii+1
Until iiii=len_string_bytes
MessageRequester("Info", Str(ElapsedMilliseconds()-time))
Debug string$


Debug "----"


time=ElapsedMilliseconds()
string$=Space(len_string_bytes*2)
iiii=0
iii = 0
Repeat
  PokeS(@string$ + iii, RSet(Hex(PeekA(*buffer+iiii)), 2, "0"))
  CompilerIf #PB_Compiler_Unicode
    iii + 4
  CompilerElse
    iii + 2
  CompilerEndIf
  iiii+1
Until iiii=len_string_bytes
MessageRequester("Info", Str(ElapsedMilliseconds()-time))
Debug string$


And switch off Debug for the real time :wink:

Bernd


Last edited by infratec on Mon Apr 18, 2016 10:27 pm, edited 1 time in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Mon Apr 18, 2016 10:25 pm 
Offline
Addict
Addict

Joined: Sat Mar 02, 2013 9:17 am
Posts: 930
@Bernd
It is not important for get the exactely time.
I must supporting sizes up to about 2e9.
Important is the result, making things tricky.

Regards werner

_________________
http://www.nachtoptik.de


Last edited by walbus on Tue Apr 19, 2016 3:31 pm, edited 1 time in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Mon Apr 18, 2016 10:28 pm 
Offline
Always Here
Always Here

Joined: Sun Sep 07, 2008 12:45 pm
Posts: 5075
Location: Germany
No,

it takes 10 seconds.

Ok, your version is faster.
But the point is not Hex() :wink:


Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Mon Apr 18, 2016 10:36 pm 
Offline
Addict
Addict

Joined: Sat Mar 02, 2013 9:17 am
Posts: 930
You must have a fast computer.
My c. run, it looks endles and the difference is very important.
The mostly users have computers same mine, i think.
Yep, you want to the moon, you can use a old rocket, or you can ask scotty for beaming.

_________________
http://www.nachtoptik.de


Last edited by walbus on Sat Apr 23, 2016 10:21 pm, edited 1 time in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Tue Apr 19, 2016 5:39 am 
Offline
PureBasic Expert
PureBasic Expert

Joined: Sun Aug 08, 2004 5:21 am
Posts: 3707
Location: Netherlands
Bin2Data (for turning a file into a DataSection), uses SSE2 to convert to hex.
viewtopic.php?f=27&t=49196
It's a very fast method for a large amount of data.

_________________
macOS 10.15 Catalina, Windows 10


Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Tue Apr 19, 2016 7:20 am 
Offline
PureBasic Expert
PureBasic Expert

Joined: Sun Aug 08, 2004 5:21 am
Posts: 3707
Location: Netherlands
Here's even faster (SSE2) :wink:
Code:
DeclareModule HexData
 
  Declare.s HexData(*mem, size, lowercase = #False)
 
EndDeclareModule

Module HexData
 
  DisableDebugger
  EnableExplicit
  EnableASM
 
  CompilerIf #PB_Compiler_Processor = #PB_Processor_x86
    Macro rax : eax : EndMacro
    Macro rcx : ecx : EndMacro
    Macro rdx : edx : EndMacro
    Macro rsp : esp : EndMacro 
  CompilerEndIf
 
  Macro M_movdqa(r1, r2)
    !movdqa r1, r2
  EndMacro
 
  Macro M_movdqu(r1, r2)
    !movdqu r1, r2
  EndMacro
 
  Procedure.s HexData(*mem, size, lowercase = #False)
    Protected Dim b.u(size + 30)
    ; init xmm registers
    !mov eax, [p.v_lowercase]
    !shl eax, 29
    !and eax, 0x20000000
    !or eax, 0x0739300f
    !movd xmm0, eax
    !punpcklbw xmm0, xmm0
    !punpcklwd xmm0, xmm0
    !pshufd xmm2, xmm0, 00000000b
    !pshufd xmm3, xmm0, 01010101b
    !pshufd xmm4, xmm0, 10101010b
    !pshufd xmm5, xmm0, 11111111b
    ; load source, destination and size
    mov rax, [p.a_b]
    mov rcx, [p.v_size]
    mov rdx, [p.p_mem]
    add rcx, rdx
    shr rdx, 4
    shl rdx, 4
    ; backup xmm6 and xmm7
    M_movdqu ([rsp - 16], xmm6)
    M_movdqu ([rsp - 32], xmm7)
    ; main loop
    !.l_hexdata_loop:
    M_movdqa (xmm0, [rdx])
    !movdqa xmm1, xmm0
    !psrld xmm0, 4
    !pand xmm0, xmm2
    !pand xmm1, xmm2
    !por xmm0, xmm3
    !por xmm1, xmm3
    !movdqa xmm6, xmm0
    !movdqa xmm7, xmm1
    !pcmpgtb xmm6, xmm4
    !pcmpgtb xmm7, xmm4
    !pand xmm6, xmm5
    !pand xmm7, xmm5
    !paddb xmm0, xmm6
    !paddb xmm7, xmm1
    !movdqa xmm1, xmm0
    !punpcklbw xmm0, xmm7
    !punpckhbw xmm1, xmm7
    M_movdqu ([rax], xmm0)
    M_movdqu ([rax + 16], xmm1)
    add rdx, 16
    add rax, 32
    cmp rdx, rcx
    !jb .l_hexdata_loop
    ; restore xmm6 and xmm7
    M_movdqu (xmm6, [rsp - 16])
    M_movdqu (xmm7, [rsp - 32])
    *mem & 15 : b(*mem + size) = 0
    ProcedureReturn PeekS(@b(*mem), -1, #PB_Ascii)
  EndProcedure
 
EndModule



; test module
A.q = $0123456789abcdef
Debug HexData::HexData(@A, 8, #True)

Dim Buffer.b(100)
RandomData(@Buffer(), 100)
Debug HexData::HexData(@Buffer(), 100, #True)



Edit:
Newer versions of PureBasic use unicode internally.
The code above creates the result in ascii and converts it to unicode when returning the result.
While this is very fast on macOS, it seems to be slower on Windows and Linux.
If you are using Windows or Linux, you might want to try the unicode version below and see if it is faster.

Code:
; Unicode only !!!

DeclareModule HexData
 
  Declare.s HexData(*mem, size, lowercase = #False)
 
EndDeclareModule

Module HexData
 
  DisableDebugger
  EnableExplicit
  EnableASM
 
  CompilerIf #PB_Compiler_Processor = #PB_Processor_x86
    Macro rax : eax : EndMacro
    Macro rcx : ecx : EndMacro
    Macro rdx : edx : EndMacro
    Macro rsp : esp : EndMacro 
  CompilerEndIf
 
  Macro M_movdqa(r1, r2)
    !movdqa r1, r2
  EndMacro
 
  Macro M_movdqu(r1, r2)
    !movdqu r1, r2
  EndMacro
 
  Procedure.s HexData(*mem, size, lowercase = #False)
    Protected Dim b.l(size + 30)
    ; init xmm registers
    !mov eax, [p.v_lowercase]
    !shl eax, 29
    !and eax, 0x20000000
    !or eax, 0x0739300f
    !movd xmm0, eax
    !punpcklbw xmm0, xmm0
    !punpcklwd xmm0, xmm0
    !pshufd xmm2, xmm0, 00000000b
    !pshufd xmm3, xmm0, 01010101b
    !pshufd xmm4, xmm0, 10101010b
    !pshufd xmm5, xmm0, 11111111b
    ; load source, destination and size
    mov rax, [p.a_b]
    mov rcx, [p.v_size]
    mov rdx, [p.p_mem]
    add rcx, rdx
    shr rdx, 4
    shl rdx, 4
    ; backup xmm6 and xmm7
    M_movdqu ([rsp - 16], xmm6)
    M_movdqu ([rsp - 32], xmm7)
    ; main loop
    !.l_hexdata_loop:
    M_movdqa (xmm0, [rdx])
    !movdqa xmm1, xmm0
    !psrld xmm0, 4
    !pand xmm0, xmm2
    !pand xmm1, xmm2
    !por xmm0, xmm3
    !por xmm1, xmm3
    !movdqa xmm6, xmm0
    !movdqa xmm7, xmm1
    !pcmpgtb xmm6, xmm4
    !pcmpgtb xmm7, xmm4
    !pand xmm6, xmm5
    !pand xmm7, xmm5
    !paddb xmm0, xmm6
    !paddb xmm7, xmm1
    !movdqa xmm1, xmm0
    !punpcklbw xmm0, xmm7
    !punpckhbw xmm1, xmm7
    !pxor xmm7, xmm7
    !movdqa xmm6, xmm0
    !punpcklbw xmm0, xmm7
    !punpckhbw xmm6, xmm7
    M_movdqu ([rax], xmm0)
    M_movdqu ([rax + 16], xmm6)
    !movdqa xmm6, xmm1
    !punpcklbw xmm1, xmm7
    !punpckhbw xmm6, xmm7
    M_movdqu ([rax + 32], xmm1)
    M_movdqu ([rax + 48], xmm6)
    add rdx, 16
    add rax, 64
    cmp rdx, rcx
    !jb .l_hexdata_loop
    ; restore xmm6 and xmm7
    M_movdqu (xmm6, [rsp - 16])
    M_movdqu (xmm7, [rsp - 32])
    *mem & 15 : b(*mem + size) = 0
    ProcedureReturn PeekS(@b(*mem))
  EndProcedure 
 
EndModule



; test module
A.q = $0123456789abcdef
Debug HexData::HexData(@A, 8, #True)

Dim Buffer.b(100)
RandomData(@Buffer(), 100)
Debug HexData::HexData(@Buffer(), 100, #True)

_________________
macOS 10.15 Catalina, Windows 10


Last edited by wilbert on Thu Nov 07, 2019 8:36 am, edited 5 times in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Tue Apr 19, 2016 7:38 am 
Offline
Addict
Addict

Joined: Sat Mar 02, 2013 9:17 am
Posts: 930
Many thanks wilbert

I'll install your code now !

_________________
http://www.nachtoptik.de


Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Tue Apr 19, 2016 12:34 pm 
Offline
PureBasic Expert
PureBasic Expert

Joined: Sun Aug 08, 2004 5:21 am
Posts: 3707
Location: Netherlands
walbus wrote:
I'll install your code now !

I updated the module code above with an additional lowercase flag.
Some people prefer lowercase, others uppercase and this way you don't have the additional time the PB conversion functions would take.

_________________
macOS 10.15 Catalina, Windows 10


Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Wed Nov 06, 2019 2:33 pm 
Offline
Addict
Addict

Joined: Thu Jun 07, 2007 3:25 pm
Posts: 3965
Location: Berlin, Germany
How was it possible that I missed that code by wilbert? :-)
Thank you very much!

_________________
Please excuse my flawed English. My native language is PureBasic.
Search
RSBasic's backups


Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Wed Nov 06, 2019 4:37 pm 
Offline
Enthusiast
Enthusiast

Joined: Tue May 26, 2009 2:11 pm
Posts: 664
Maybe I'm wrong, but shouldn't
Code:
A.q = $0123456789abcdef
Debug HexData::HexData(@A, 8, #True)
give "0123456789abcdef"

and
Code:
A.q = $0123456789abcdef
Debug HexData::HexData(@A, 16, #True)
return "00000000000000000123456789abcdef"
instead of
"efcdab89674523011700000000000000"?

_________________
Image


Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Wed Nov 06, 2019 4:44 pm 
Offline
PureBasic Expert
PureBasic Expert

Joined: Sun Aug 08, 2004 5:21 am
Posts: 3707
Location: Netherlands
Lord wrote:
Maybe I'm wrong, but shouldn't
Code:
A.q = $0123456789abcdef
Debug HexData::HexData(@A, 8, #True)
give "0123456789abcdef"

No, Intel is little-endian. The least significant bytes are stored first.
So a 64 bits integer $0123456789abcdef is stored in memory as
ef cd ab 89 67 45 23 01

This procedure doesn't return a hex value but returns a string of all bytes of a memory area as they are stored in memory.

_________________
macOS 10.15 Catalina, Windows 10


Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Wed Nov 06, 2019 7:58 pm 
Offline
Enthusiast
Enthusiast

Joined: Tue May 26, 2009 2:11 pm
Posts: 664
wilbert wrote:
...
This procedure doesn't return a hex value but returns a string of all bytes of a memory area as they are stored in memory.
Oh, I see.
I must have missed the exit from hex-string-highway to hex-data-highway. :wink:

_________________
Image


Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Thu Nov 07, 2019 8:38 am 
Offline
PureBasic Expert
PureBasic Expert

Joined: Sun Aug 08, 2004 5:21 am
Posts: 3707
Location: Netherlands
I updated my module code above by adding a unicode only version which might be faster on Windows and Linux.

_________________
macOS 10.15 Catalina, Windows 10


Top
 Profile  
Reply with quote  
 Post subject: Re: HEX string generating - extreme fast
PostPosted: Thu Nov 07, 2019 3:34 pm 
Offline
Enthusiast
Enthusiast

Joined: Thu May 21, 2009 6:56 pm
Posts: 592
Hi Wilbert

I tested your newer unicode version in PB 5.71 Windows and the time for 100,000 iterations was 17ms so it is faster than the previous version by 4ms and about 30 times faster then my PB code. So thank-you very much.

I would be helpful to have a PokeH() function as well that takes a Hex string and converts it to the binary ascii values. :)

Simon

_________________
Simon White
dCipher Computing


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 18 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

 


Powered by phpBB © 2008 phpBB Group
subSilver+ theme by Canver Software, sponsor Sanal Modifiye