Page 2 of 2

Re: SHA3 Module (Keccak Module)

Posted: Mon Sep 09, 2013 4:53 pm
by walbus
Yep, many sorry, i have oversight ...

i have set the module in my Multifunktion cryptographie and steganographie Tool QUICK-AES-256

It works fine !

Many thanks for this great work !

http://www.quick-aes-256.de

Re: SHA3 Module (Keccak Module)

Posted: Tue Sep 10, 2013 8:22 am
by sec
Version: 1.4.7
- optimize the 64 bit code by wilbert

Re: SHA3 Module (Keccak Module)

Posted: Thu Sep 12, 2013 3:39 am
by sec
Version: 1.4.8
- another update improving speed by wilbert

Re: SHA3 Module (Keccak Module)

Posted: Sun May 18, 2014 7:21 am
by coco2
Very nice work

Re: SHA3 Module (Keccak Module)

Posted: Tue May 12, 2015 7:34 am
by wilbert
In time a change was made to the SHA3 draft from the original Keccak version.
To comply to the final FIPS202 you have to make a change to the finalize procedure.

Change

Code: Select all

*st\a[*ctx\pos] ! 1
into

Code: Select all

*st\a[*ctx\pos] ! %110
This change should bring the output in line with the most recent test vectors. :)
For Shake128 and Shake256 the xor value should be %11111

Re: SHA3 Module (Keccak Module)

Posted: Tue May 12, 2015 11:50 am
by wilbert
I noticed I still had a file I did a long time ago but never posted.
It's an optimized 64 bit SHA3 module in case you don't need 32 bit support.
I updated that also to be compatible to final FIPS202 specification

Code: Select all

;======================================================================
; Module:          SHA3_x64.pbi
;
; Author:          Wilbert, Sec
; Date:            Nov 21, 2015
; Version:         1.0.3
; Target Compiler: PureBasic 5.20+
; Target OS:       All (x64)
; License:         Free, unrestricted, no warranty whatsoever
;                  Use at your own risk
;======================================================================

DeclareModule SHA3
  
  #ModeKeccak = 2048
  
  Structure SHA3Context
    state.q[25]       ; offset 0
    tmpstate.q[25]    ; offset 200
    pos.a             ; offset 400
    rsiz.a            ; offset 401
    hsiz.a            ; offset 402
    xor_byte.a        ; offset 403
  EndStructure 
  
  ;-Exported functions
  Declare$ SHA3Fingerprint(*Buffer, Size, Mode=224)   
  Declare$ SHA3FileFingerprint(Filename$, Mode=224, Offset.q=0, Length.q=0, ProgressBarGadgetNr=-1)
  Declare SHA3_init(*ctx.SHA3Context, hsiz.i)
  Declare SHA3_update(*ctx.SHA3Context, *Data, len.i)
  Declare$ SHA3_final(*ctx.SHA3Context)
  
EndDeclareModule

Module SHA3
  EnableExplicit
  
  Structure WordArray
    w.w[0]
  EndStructure
  
  Structure AArray
    a.a[0]
  EndStructure       
  
  DataSection
    _hex_bytes:
    !db '000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f'
    !db '202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f'
    !db '404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f'
    !db '606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f'
    !db '808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9f'
    !db 'a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf'
    !db 'c0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedf'
    !db 'e0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff'
  EndDataSection
  
  Macro MovX(r1, r2) ; move 64 bits between r64 and xmm registers
      !movq r1, r2
  EndMacro
    
  Macro PrepareC(xmm_reg, x)
    !mov r8, [rdx + x * 8]
    !mov r9, [rdx + x * 8 + 40]
    !mov r10, [rdx + x * 8 + 80]
    !mov r11, [rdx + x * 8 + 120]
    !mov r12, [rdx + x * 8 + 160]
    !xor r8, r9
    !xor r8, r10
    !xor r8, r11
    !xor r8, r12    
    MovX(xmm_reg, r8)
  EndMacro
  
  Macro D(reg, xmm_reg1, xmm_reg2)
    MovX(reg, xmm_reg1)
    MovX(rax, xmm_reg2)
    !rol rax, 1
    !xor reg, rax
  EndMacro
  
  Macro B(xmm_reg, x, y, d, r)
    !mov rax, [rdx + x * 8 + y * 40]
    !xor rax, d
    CompilerIf r <> 0
      !rol rax, r
    CompilerEndIf
    MovX(xmm_reg, rax)
  EndMacro
  
  Macro E_C(xmm_reg, x, y, b0, b1, b2, rnd, rndc = 0)
    !movq xmm10, b1
    !pandn xmm10, b2
    !pxor xmm10, b0
    CompilerIf x = 0 And y = 0
      !mov rax, rndc
      MovX(xmm_reg, rax)
      !pxor xmm10, xmm_reg
    CompilerEndIf
    CompilerIf rnd < 23
      CompilerIf y = 0
        !movq xmm_reg, xmm10
      CompilerElse
        !pxor xmm_reg, xmm10
      CompilerEndIf  
    CompilerEndIf
    !movq [rcx + x * 8 + y * 40], xmm10
  EndMacro
  
  Macro SHA3Round(rnd, rndc)
    ; Prepare C[.] (round 1 only)
    CompilerIf rnd = 0
      PrepareC(xmm5, 0)
      PrepareC(xmm6, 1)
      PrepareC(xmm7, 2)
      PrepareC(xmm8, 3)
      PrepareC(xmm9, 4)
    CompilerEndIf
    
    ; D[.]
    D( r8, xmm9, xmm6)
    D( r9, xmm5, xmm7)
    D(r10, xmm6, xmm8)
    D(r11, xmm7, xmm9)
    D(r12, xmm8, xmm5)
    
    ; B[.] (y = 0)  
    B(xmm0, 0, 0,  r8,  0)
    B(xmm1, 1, 1,  r9, 44)
    B(xmm2, 2, 2, r10, 43)
    B(xmm3, 3, 3, r11, 21)
    B(xmm4, 4, 4, r12, 14)
    ; E[.,.] and C[.]
    E_C(xmm5, 0, 0, xmm0, xmm1, xmm2, rnd, rndc)  
    E_C(xmm6, 1, 0, xmm1, xmm2, xmm3, rnd)  
    E_C(xmm7, 2, 0, xmm2, xmm3, xmm4, rnd)  
    E_C(xmm8, 3, 0, xmm3, xmm4, xmm0, rnd)  
    E_C(xmm9, 4, 0, xmm4, xmm0, xmm1, rnd)
    
    ; B[.] (y = 1)  
    B(xmm0, 3, 0, r11, 28)
    B(xmm1, 4, 1, r12, 20)
    B(xmm2, 0, 2,  r8,  3)
    B(xmm3, 1, 3,  r9, 45)
    B(xmm4, 2, 4, r10, 61)
    ; E[.,.] and C[.]
    E_C(xmm5, 0, 1, xmm0, xmm1, xmm2, rnd)  
    E_C(xmm6, 1, 1, xmm1, xmm2, xmm3, rnd)  
    E_C(xmm7, 2, 1, xmm2, xmm3, xmm4, rnd)  
    E_C(xmm8, 3, 1, xmm3, xmm4, xmm0, rnd)  
    E_C(xmm9, 4, 1, xmm4, xmm0, xmm1, rnd)
    
    ; B[.] (y = 2)  
    B(xmm0, 1, 0,  r9,  1)
    B(xmm1, 2, 1, r10,  6)
    B(xmm2, 3, 2, r11, 25)
    B(xmm3, 4, 3, r12,  8)
    B(xmm4, 0, 4,  r8, 18)
    ; E[.,.] and C[.]
    E_C(xmm5, 0, 2, xmm0, xmm1, xmm2, rnd)  
    E_C(xmm6, 1, 2, xmm1, xmm2, xmm3, rnd)  
    E_C(xmm7, 2, 2, xmm2, xmm3, xmm4, rnd)  
    E_C(xmm8, 3, 2, xmm3, xmm4, xmm0, rnd)  
    E_C(xmm9, 4, 2, xmm4, xmm0, xmm1, rnd)
    
    ; B[.] (y = 3)  
    B(xmm0, 4, 0, r12, 27)
    B(xmm1, 0, 1,  r8, 36)
    B(xmm2, 1, 2,  r9, 10)
    B(xmm3, 2, 3, r10, 15)
    B(xmm4, 3, 4, r11, 56)
    ; E[.,.] and C[.]
    E_C(xmm5, 0, 3, xmm0, xmm1, xmm2, rnd)  
    E_C(xmm6, 1, 3, xmm1, xmm2, xmm3, rnd)  
    E_C(xmm7, 2, 3, xmm2, xmm3, xmm4, rnd)  
    E_C(xmm8, 3, 3, xmm3, xmm4, xmm0, rnd)  
    E_C(xmm9, 4, 3, xmm4, xmm0, xmm1, rnd)
    
    ; B[.] (y = 4)  
    B(xmm0, 2, 0, r10, 62)
    B(xmm1, 3, 1, r11, 55)
    B(xmm2, 4, 2, r12, 39)
    B(xmm3, 0, 3,  r8, 41)
    B(xmm4, 1, 4,  r9,  2)
    ; E[.,.] and C[.]
    E_C(xmm5, 0, 4, xmm0, xmm1, xmm2, rnd)  
    E_C(xmm6, 1, 4, xmm1, xmm2, xmm3, rnd)  
    E_C(xmm7, 2, 4, xmm2, xmm3, xmm4, rnd)  
    E_C(xmm8, 3, 4, xmm3, xmm4, xmm0, rnd)  
    E_C(xmm9, 4, 4, xmm4, xmm0, xmm1, rnd)
    
    !xchg rdx, rcx
  EndMacro
  
  Procedure SHA3_Process(*ctx.SHA3Context)
    !mov rdx, [p.p_ctx]
    !lea rcx, [rdx + 200]
    
    ; store volatile registers
    !sub rsp, 96
    
    !movdqu [rsp], xmm6
    !movdqu [rsp + 16], xmm7
    !movdqu [rsp + 32], xmm8
    !movdqu [rsp + 48], xmm9
    !movdqu [rsp + 64], xmm10
    !movdqu [rsp + 80], xmm11
    !push r12
    
    ; process rounds
    SHA3Round( 0, 0x0000000000000001)
    SHA3Round( 1, 0x0000000000008082)
    SHA3Round( 2, 0x800000000000808A)
    SHA3Round( 3, 0x8000000080008000)
    SHA3Round( 4, 0x000000000000808B)
    SHA3Round( 5, 0x0000000080000001)
    SHA3Round( 6, 0x8000000080008081)
    SHA3Round( 7, 0x8000000000008009)
    SHA3Round( 8, 0x000000000000008A)
    SHA3Round( 9, 0x0000000000000088)
    SHA3Round(10, 0x0000000080008009)
    SHA3Round(11, 0x000000008000000A)
    SHA3Round(12, 0x000000008000808B)
    SHA3Round(13, 0x800000000000008B)
    SHA3Round(14, 0x8000000000008089)
    SHA3Round(15, 0x8000000000008003)
    SHA3Round(16, 0x8000000000008002)
    SHA3Round(17, 0x8000000000000080)
    SHA3Round(18, 0x000000000000800A)
    SHA3Round(19, 0x800000008000000A)
    SHA3Round(20, 0x8000000080008081)
    SHA3Round(21, 0x8000000000008080)
    SHA3Round(22, 0x0000000080000001)
    SHA3Round(23, 0x8000000080008008)
    
    ; restore volatile registers
    !pop r12
    !movdqu xmm6, [rsp]
    !movdqu xmm7, [rsp + 16]
    !movdqu xmm8, [rsp + 32]
    !movdqu xmm9, [rsp + 48]
    !movdqu xmm10, [rsp + 64]
    !movdqu xmm11, [rsp + 80]
    !add rsp, 96
  EndProcedure  
  
  Procedure SHA3_init(*ctx.SHA3Context, hsiz.i)
    FillMemory(*ctx, 200)
    *ctx\pos = 0
    If hsiz & #ModeKeccak
      *ctx\xor_byte = 1
    Else
      *ctx\xor_byte = %110
    EndIf
    Select hsiz & 1023
      Case 512
        *ctx\hsiz = 64
        *ctx\rsiz = 72
      Case 384
        *ctx\hsiz = 48
        *ctx\rsiz = 104
      Case 256
        *ctx\hsiz = 32
        *ctx\rsiz = 136
      Default; 224
        *ctx\hsiz = 28
        *ctx\rsiz = 144
    EndSelect
  EndProcedure
  
  Procedure SHA3_update(*ctx.SHA3Context, *Data, len.i)
    While len
      
      ; absorb data setup
      !mov rdx, [p.p_ctx]
      !mov r8, [p.v_len]
      !movzx r11, byte [rdx + 401]  ; *ctx\rsiz
      !movzx r10, byte [rdx + 400]  ; *ctx\pos
      !mov rcx, r11
      !sub rcx, r10                 ; rcx = number of bytes to absorb
      !cmp r8, rcx
      !cmovl rcx, r8                ; if len < absorb, absorb = len
      !mov r8, [p.p_Data]           ; r8 = source
      !lea r9, [rdx + r10]          ; r9 = destination
      !add [p.p_Data], rcx          ; *Data + absorb
      !sub [p.v_len], rcx           ; len - absorb
      !add r10, rcx                 ; *ctx\pos + absorb
      
      ; actual xor absorb
      !sub rcx, 8
      !jc sha3.l_update_l1
      !sha3.l_update_l0:
      !mov rax, [r8 + rcx]
      !xor [r9 + rcx], rax
      !sub rcx, 8
      !jnc sha3.l_update_l0
      !sha3.l_update_l1:
      !add rcx, 8
      !jz sha3.l_update_l3
      !sub rcx, 1
      !sha3.l_update_l2:
      !mov al, [r8 + rcx]
      !xor [r9 + rcx], al
      !sub rcx, 1
      !jnc sha3.l_update_l2
      !sha3.l_update_l3:      
      
      ; process absorbed data if *ctx\pos = *ctx\rsiz
      !xor rax, rax
      !cmp r10, r11
      !cmove r10, rax
      !mov [rdx + 400], r10b
      !jne sha3.l_update_l4
      SHA3_Process(*ctx)
      !sha3.l_update_l4:
      
    Wend
  EndProcedure
  
  Procedure$ SHA3_final(*ctx.SHA3Context)
    Dim hash.w(*ctx\hsiz)
    Define *hex_bytes.WordArray = ?_hex_bytes
    Define *st.AArray = *ctx
    Define.i i
    
    *st\a[*ctx\pos] ! *ctx\xor_byte
    *st\a[*ctx\rsiz - 1] ! $80
    SHA3_Process(*ctx)
    
    While i < *ctx\hsiz
      hash(i) = *hex_bytes\w[*st\a[i]]
      i + 1
    Wend
    ProcedureReturn PeekS(@hash(), -1, #PB_Ascii)
  EndProcedure
  
  
  ;-Exported functions
  Procedure$ SHA3FileFingerprint(Filename$, Mode=224, Offset.q=0, Length.q=0, ProgressBarGadgetNr=-1)
    
    #gnChunkSize = 1024 * 32; 32 KiB
    Dim DataChunk.q(#gnChunkSize >> 3)
    
    Define hash.s, file.i, error.i, ctx.SHA3Context
    Define.q nBytes, nTotalBytes, nTotalBytesRead
    
    file = ReadFile(#PB_Any, Filename$)
    If file
      nTotalBytes = Lof(file)
      If Offset < nTotalBytes
        FileSeek(file, Offset)
        If Length = 0
          Length = nTotalBytes
        EndIf
        If Offset + Length > nTotalBytes
          Length = nTotalBytes - Offset
        EndIf
        nTotalBytes = Length
        
        SHA3_init(@ctx, Mode)
        
        While Length > 0
          nBytes = ReadData(file, @DataChunk(), #gnChunkSize)
          If nBytes
            If nBytes > Length
              nBytes = Length
            EndIf
            SHA3_update(@ctx, @DataChunk(), nBytes)
            Length - nBytes
            nTotalBytesRead + nBytes
            If ProgressBarGadgetNr >= 0 And IsGadget(ProgressBarGadgetNr)
              SetGadgetState(ProgressBarGadgetNr, 100 * nTotalBytesRead / nTotalBytes)
            EndIf
          Else
            error = #True
            Break
          EndIf
        Wend
        
        If error <> #True
          hash = SHA3_final(@ctx)
        EndIf
        
      EndIf
      CloseFile(file)
    EndIf
    
    ProcedureReturn hash
  EndProcedure
  
  Procedure$ SHA3Fingerprint(*Buffer, Size, Mode=224)             
    Define ctx.SHA3Context
    SHA3_init(@ctx, Mode)
    SHA3_update(@ctx, *Buffer, Size)
    ProcedureReturn SHA3_final(@ctx)   
  EndProcedure
  
EndModule

Re: SHA3 Module (Keccak Module)

Posted: Tue May 12, 2015 1:52 pm
by Kukulkan
Really nice, thank you!

Re: SHA3 Module (Keccak Module)

Posted: Tue May 12, 2015 5:44 pm
by wilbert
Kukulkan wrote:Really nice, thank you!
Thanks :)

I updated the post above with a small change.

Code: Select all

Debug SHA3::Fingerprint(@"", 0, 256)
will output the hash from the Draft FIPS202 specification.

Code: Select all

Debug SHA3::Fingerprint(@"", 0, 256 | SHA3::#ModeKeccak)
will output the original Keccak hash.

It was little effort to support both versions this way.

I'm thinking of adding Shake128 and Shake256 as well.
Those are also in the draft specification and are functions with a variable digest size.
You could get a 32 bit hash but also a 4096 bit hash using those functions.
The only thing is that I don't know how to add them without breaking backward compatibility since an extra parameter would be needed (digest size).

Re: SHA3 Module (Keccak Module)

Posted: Fri Aug 07, 2015 12:21 pm
by wilbert

Re: SHA3 Module (Keccak Module)

Posted: Sun Jun 05, 2016 5:07 pm
by Etayson
wilbert wrote:
Kukulkan wrote:Really nice, thank you!
Thanks :)

I updated the post above with a small change.

Code: Select all

Debug SHA3::Fingerprint(@"", 0, 256)
will output the hash from the Draft FIPS202 specification.

Code: Select all

Debug SHA3::Fingerprint(@"", 0, 256 | SHA3::#ModeKeccak)
will output the original Keccak hash.

It was little effort to support both versions this way.

I'm thinking of adding Shake128 and Shake256 as well.
Those are also in the draft specification and are functions with a variable digest size.
You could get a 32 bit hash but also a 4096 bit hash using those functions.
The only thing is that I don't know how to add them without breaking backward compatibility since an extra parameter would be needed (digest size).
hi, when i use:
IncludeFile "SHA3_x64.pbi"
Debug SHA3::Fingerprint(@"testing", 0, 256 | SHA3::#ModeKeccak)

I got error: Module item'Fingerprint' not found

Why i get this error&

Re: SHA3 Module (Keccak Module)

Posted: Sun Jun 05, 2016 5:21 pm
by infratec
That's right.

You have to rename the procedures and the declares from

SHA3Fingerprint

to

Fingerprint

Bernd

Re: SHA3 Module (Keccak Module)

Posted: Sun Jun 05, 2016 7:37 pm
by walbus
Wilberts codes are "Arts of work" !

Wilbert is NOT a babbler, you have a problem and ask friendly, wilbert helps !

For the SHA3 Module,
rename simple inside the module with the PB rename function, as sample so :

SHA3::Fingerprint_(*Buffer, Size, 256)

Re: SHA3 Module (Keccak Module)

Posted: Sun Jun 05, 2016 8:14 pm
by RichAlgeni
walbus wrote:Wilberts codes are "Arts of work" !

Wilbert is NOT a babbler, you have a problem and ask friendly, wilbert helps !

For the SHA3 Module,
rename simple inside the module with the PB rename function, as sample so :

SHA3::Fingerprint_(*Buffer, Size, 256)
Here, here!!

Re: SHA3 Module (Keccak Module)

Posted: Sun Jun 26, 2016 5:25 pm
by Etayson
Hello all . I really need help with the module sha-3 kessak. I'm writing a program for mining Ethereum . But Sha-3 module provides the result is not as it should be.
Can someone suggest how to fix the module to the results were the same ?
Here are quotes from ethash doc (https://github.com/ethereum/wiki/wiki/Ethash)

A note regarding "SHA3" hashes described in this specification
Ethereum's development coincided with the development of the SHA3 standard, and the standards process made a late change in the padding of the finalized hash algorithm, so that Ethereum's "sha3_256" and "sha3_512" hashes are not standard sha3 hashes, but a variant often referred to as "Keccak-256" and "Keccak-512" in other contexts. See discussion, e.g. here, here, or here.

Please keep that in mind as "sha3" hashes are referred to in the description of the algorithm below.


And here is a link to sha-3 kessak used in Ethash: https://github.com/ethereum/ethash/blob ... ash/sha3.c