SHA3 Module (Keccak Module)

walbus · Post by **walbus** » Mon Sep 09, 2013 4:53 pm

Yep, many sorry, i have oversight ...

i have set the module in my Multifunktion cryptographie and steganographie Tool QUICK-AES-256

It works fine !

Many thanks for this great work !

http://www.quick-aes-256.de

sec · Post by **sec** » Tue Sep 10, 2013 8:22 am

Version: 1.4.7
- optimize the 64 bit code by wilbert

sec · Post by **sec** » Thu Sep 12, 2013 3:39 am

Version: 1.4.8
- another update improving speed by wilbert

coco2 · Post by **coco2** » Sun May 18, 2014 7:21 am

Very nice work

wilbert · Post by **wilbert** » Tue May 12, 2015 7:34 am

In time a change was made to the SHA3 draft from the original Keccak version.
To comply to the final FIPS202 you have to make a change to the finalize procedure.

Change

Code: Select all

*st\a[*ctx\pos] ! 1

into

Code: Select all

*st\a[*ctx\pos] ! %110

This change should bring the output in line with the most recent test vectors.

For Shake128 and Shake256 the xor value should be %11111

wilbert · Post by **wilbert** » Tue May 12, 2015 11:50 am

I noticed I still had a file I did a long time ago but never posted.
It's an optimized 64 bit SHA3 module in case you don't need 32 bit support.
I updated that also to be compatible to final FIPS202 specification

Code: Select all

;======================================================================
; Module:          SHA3_x64.pbi
;
; Author:          Wilbert, Sec
; Date:            Nov 21, 2015
; Version:         1.0.3
; Target Compiler: PureBasic 5.20+
; Target OS:       All (x64)
; License:         Free, unrestricted, no warranty whatsoever
;                  Use at your own risk
;======================================================================

DeclareModule SHA3
  
  #ModeKeccak = 2048
  
  Structure SHA3Context
    state.q[25]       ; offset 0
    tmpstate.q[25]    ; offset 200
    pos.a             ; offset 400
    rsiz.a            ; offset 401
    hsiz.a            ; offset 402
    xor_byte.a        ; offset 403
  EndStructure 
  
  ;-Exported functions
  Declare$ SHA3Fingerprint(*Buffer, Size, Mode=224)   
  Declare$ SHA3FileFingerprint(Filename$, Mode=224, Offset.q=0, Length.q=0, ProgressBarGadgetNr=-1)
  Declare SHA3_init(*ctx.SHA3Context, hsiz.i)
  Declare SHA3_update(*ctx.SHA3Context, *Data, len.i)
  Declare$ SHA3_final(*ctx.SHA3Context)
  
EndDeclareModule

Module SHA3
  EnableExplicit
  
  Structure WordArray
    w.w[0]
  EndStructure
  
  Structure AArray
    a.a[0]
  EndStructure       
  
  DataSection
    _hex_bytes:
    !db '000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f'
    !db '202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f'
    !db '404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f'
    !db '606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f'
    !db '808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9f'
    !db 'a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf'
    !db 'c0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedf'
    !db 'e0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff'
  EndDataSection
  
  Macro MovX(r1, r2) ; move 64 bits between r64 and xmm registers
      !movq r1, r2
  EndMacro
    
  Macro PrepareC(xmm_reg, x)
    !mov r8, [rdx + x * 8]
    !mov r9, [rdx + x * 8 + 40]
    !mov r10, [rdx + x * 8 + 80]
    !mov r11, [rdx + x * 8 + 120]
    !mov r12, [rdx + x * 8 + 160]
    !xor r8, r9
    !xor r8, r10
    !xor r8, r11
    !xor r8, r12    
    MovX(xmm_reg, r8)
  EndMacro
  
  Macro D(reg, xmm_reg1, xmm_reg2)
    MovX(reg, xmm_reg1)
    MovX(rax, xmm_reg2)
    !rol rax, 1
    !xor reg, rax
  EndMacro
  
  Macro B(xmm_reg, x, y, d, r)
    !mov rax, [rdx + x * 8 + y * 40]
    !xor rax, d
    CompilerIf r <> 0
      !rol rax, r
    CompilerEndIf
    MovX(xmm_reg, rax)
  EndMacro
  
  Macro E_C(xmm_reg, x, y, b0, b1, b2, rnd, rndc = 0)
    !movq xmm10, b1
    !pandn xmm10, b2
    !pxor xmm10, b0
    CompilerIf x = 0 And y = 0
      !mov rax, rndc
      MovX(xmm_reg, rax)
      !pxor xmm10, xmm_reg
    CompilerEndIf
    CompilerIf rnd < 23
      CompilerIf y = 0
        !movq xmm_reg, xmm10
      CompilerElse
        !pxor xmm_reg, xmm10
      CompilerEndIf  
    CompilerEndIf
    !movq [rcx + x * 8 + y * 40], xmm10
  EndMacro
  
  Macro SHA3Round(rnd, rndc)
    ; Prepare C[.] (round 1 only)
    CompilerIf rnd = 0
      PrepareC(xmm5, 0)
      PrepareC(xmm6, 1)
      PrepareC(xmm7, 2)
      PrepareC(xmm8, 3)
      PrepareC(xmm9, 4)
    CompilerEndIf
    
    ; D[.]
    D( r8, xmm9, xmm6)
    D( r9, xmm5, xmm7)
    D(r10, xmm6, xmm8)
    D(r11, xmm7, xmm9)
    D(r12, xmm8, xmm5)
    
    ; B[.] (y = 0)  
    B(xmm0, 0, 0,  r8,  0)
    B(xmm1, 1, 1,  r9, 44)
    B(xmm2, 2, 2, r10, 43)
    B(xmm3, 3, 3, r11, 21)
    B(xmm4, 4, 4, r12, 14)
    ; E[.,.] and C[.]
    E_C(xmm5, 0, 0, xmm0, xmm1, xmm2, rnd, rndc)  
    E_C(xmm6, 1, 0, xmm1, xmm2, xmm3, rnd)  
    E_C(xmm7, 2, 0, xmm2, xmm3, xmm4, rnd)  
    E_C(xmm8, 3, 0, xmm3, xmm4, xmm0, rnd)  
    E_C(xmm9, 4, 0, xmm4, xmm0, xmm1, rnd)
    
    ; B[.] (y = 1)  
    B(xmm0, 3, 0, r11, 28)
    B(xmm1, 4, 1, r12, 20)
    B(xmm2, 0, 2,  r8,  3)
    B(xmm3, 1, 3,  r9, 45)
    B(xmm4, 2, 4, r10, 61)
    ; E[.,.] and C[.]
    E_C(xmm5, 0, 1, xmm0, xmm1, xmm2, rnd)  
    E_C(xmm6, 1, 1, xmm1, xmm2, xmm3, rnd)  
    E_C(xmm7, 2, 1, xmm2, xmm3, xmm4, rnd)  
    E_C(xmm8, 3, 1, xmm3, xmm4, xmm0, rnd)  
    E_C(xmm9, 4, 1, xmm4, xmm0, xmm1, rnd)
    
    ; B[.] (y = 2)  
    B(xmm0, 1, 0,  r9,  1)
    B(xmm1, 2, 1, r10,  6)
    B(xmm2, 3, 2, r11, 25)
    B(xmm3, 4, 3, r12,  8)
    B(xmm4, 0, 4,  r8, 18)
    ; E[.,.] and C[.]
    E_C(xmm5, 0, 2, xmm0, xmm1, xmm2, rnd)  
    E_C(xmm6, 1, 2, xmm1, xmm2, xmm3, rnd)  
    E_C(xmm7, 2, 2, xmm2, xmm3, xmm4, rnd)  
    E_C(xmm8, 3, 2, xmm3, xmm4, xmm0, rnd)  
    E_C(xmm9, 4, 2, xmm4, xmm0, xmm1, rnd)
    
    ; B[.] (y = 3)  
    B(xmm0, 4, 0, r12, 27)
    B(xmm1, 0, 1,  r8, 36)
    B(xmm2, 1, 2,  r9, 10)
    B(xmm3, 2, 3, r10, 15)
    B(xmm4, 3, 4, r11, 56)
    ; E[.,.] and C[.]
    E_C(xmm5, 0, 3, xmm0, xmm1, xmm2, rnd)  
    E_C(xmm6, 1, 3, xmm1, xmm2, xmm3, rnd)  
    E_C(xmm7, 2, 3, xmm2, xmm3, xmm4, rnd)  
    E_C(xmm8, 3, 3, xmm3, xmm4, xmm0, rnd)  
    E_C(xmm9, 4, 3, xmm4, xmm0, xmm1, rnd)
    
    ; B[.] (y = 4)  
    B(xmm0, 2, 0, r10, 62)
    B(xmm1, 3, 1, r11, 55)
    B(xmm2, 4, 2, r12, 39)
    B(xmm3, 0, 3,  r8, 41)
    B(xmm4, 1, 4,  r9,  2)
    ; E[.,.] and C[.]
    E_C(xmm5, 0, 4, xmm0, xmm1, xmm2, rnd)  
    E_C(xmm6, 1, 4, xmm1, xmm2, xmm3, rnd)  
    E_C(xmm7, 2, 4, xmm2, xmm3, xmm4, rnd)  
    E_C(xmm8, 3, 4, xmm3, xmm4, xmm0, rnd)  
    E_C(xmm9, 4, 4, xmm4, xmm0, xmm1, rnd)
    
    !xchg rdx, rcx
  EndMacro
  
  Procedure SHA3_Process(*ctx.SHA3Context)
    !mov rdx, [p.p_ctx]
    !lea rcx, [rdx + 200]
    
    ; store volatile registers
    !sub rsp, 96
    
    !movdqu [rsp], xmm6
    !movdqu [rsp + 16], xmm7
    !movdqu [rsp + 32], xmm8
    !movdqu [rsp + 48], xmm9
    !movdqu [rsp + 64], xmm10
    !movdqu [rsp + 80], xmm11
    !push r12
    
    ; process rounds
    SHA3Round( 0, 0x0000000000000001)
    SHA3Round( 1, 0x0000000000008082)
    SHA3Round( 2, 0x800000000000808A)
    SHA3Round( 3, 0x8000000080008000)
    SHA3Round( 4, 0x000000000000808B)
    SHA3Round( 5, 0x0000000080000001)
    SHA3Round( 6, 0x8000000080008081)
    SHA3Round( 7, 0x8000000000008009)
    SHA3Round( 8, 0x000000000000008A)
    SHA3Round( 9, 0x0000000000000088)
    SHA3Round(10, 0x0000000080008009)
    SHA3Round(11, 0x000000008000000A)
    SHA3Round(12, 0x000000008000808B)
    SHA3Round(13, 0x800000000000008B)
    SHA3Round(14, 0x8000000000008089)
    SHA3Round(15, 0x8000000000008003)
    SHA3Round(16, 0x8000000000008002)
    SHA3Round(17, 0x8000000000000080)
    SHA3Round(18, 0x000000000000800A)
    SHA3Round(19, 0x800000008000000A)
    SHA3Round(20, 0x8000000080008081)
    SHA3Round(21, 0x8000000000008080)
    SHA3Round(22, 0x0000000080000001)
    SHA3Round(23, 0x8000000080008008)
    
    ; restore volatile registers
    !pop r12
    !movdqu xmm6, [rsp]
    !movdqu xmm7, [rsp + 16]
    !movdqu xmm8, [rsp + 32]
    !movdqu xmm9, [rsp + 48]
    !movdqu xmm10, [rsp + 64]
    !movdqu xmm11, [rsp + 80]
    !add rsp, 96
  EndProcedure  
  
  Procedure SHA3_init(*ctx.SHA3Context, hsiz.i)
    FillMemory(*ctx, 200)
    *ctx\pos = 0
    If hsiz & #ModeKeccak
      *ctx\xor_byte = 1
    Else
      *ctx\xor_byte = %110
    EndIf
    Select hsiz & 1023
      Case 512
        *ctx\hsiz = 64
        *ctx\rsiz = 72
      Case 384
        *ctx\hsiz = 48
        *ctx\rsiz = 104
      Case 256
        *ctx\hsiz = 32
        *ctx\rsiz = 136
      Default; 224
        *ctx\hsiz = 28
        *ctx\rsiz = 144
    EndSelect
  EndProcedure
  
  Procedure SHA3_update(*ctx.SHA3Context, *Data, len.i)
    While len
      
      ; absorb data setup
      !mov rdx, [p.p_ctx]
      !mov r8, [p.v_len]
      !movzx r11, byte [rdx + 401]  ; *ctx\rsiz
      !movzx r10, byte [rdx + 400]  ; *ctx\pos
      !mov rcx, r11
      !sub rcx, r10                 ; rcx = number of bytes to absorb
      !cmp r8, rcx
      !cmovl rcx, r8                ; if len < absorb, absorb = len
      !mov r8, [p.p_Data]           ; r8 = source
      !lea r9, [rdx + r10]          ; r9 = destination
      !add [p.p_Data], rcx          ; *Data + absorb
      !sub [p.v_len], rcx           ; len - absorb
      !add r10, rcx                 ; *ctx\pos + absorb
      
      ; actual xor absorb
      !sub rcx, 8
      !jc sha3.l_update_l1
      !sha3.l_update_l0:
      !mov rax, [r8 + rcx]
      !xor [r9 + rcx], rax
      !sub rcx, 8
      !jnc sha3.l_update_l0
      !sha3.l_update_l1:
      !add rcx, 8
      !jz sha3.l_update_l3
      !sub rcx, 1
      !sha3.l_update_l2:
      !mov al, [r8 + rcx]
      !xor [r9 + rcx], al
      !sub rcx, 1
      !jnc sha3.l_update_l2
      !sha3.l_update_l3:      
      
      ; process absorbed data if *ctx\pos = *ctx\rsiz
      !xor rax, rax
      !cmp r10, r11
      !cmove r10, rax
      !mov [rdx + 400], r10b
      !jne sha3.l_update_l4
      SHA3_Process(*ctx)
      !sha3.l_update_l4:
      
    Wend
  EndProcedure
  
  Procedure$ SHA3_final(*ctx.SHA3Context)
    Dim hash.w(*ctx\hsiz)
    Define *hex_bytes.WordArray = ?_hex_bytes
    Define *st.AArray = *ctx
    Define.i i
    
    *st\a[*ctx\pos] ! *ctx\xor_byte
    *st\a[*ctx\rsiz - 1] ! $80
    SHA3_Process(*ctx)
    
    While i < *ctx\hsiz
      hash(i) = *hex_bytes\w[*st\a[i]]
      i + 1
    Wend
    ProcedureReturn PeekS(@hash(), -1, #PB_Ascii)
  EndProcedure
  
  
  ;-Exported functions
  Procedure$ SHA3FileFingerprint(Filename$, Mode=224, Offset.q=0, Length.q=0, ProgressBarGadgetNr=-1)
    
    #gnChunkSize = 1024 * 32; 32 KiB
    Dim DataChunk.q(#gnChunkSize >> 3)
    
    Define hash.s, file.i, error.i, ctx.SHA3Context
    Define.q nBytes, nTotalBytes, nTotalBytesRead
    
    file = ReadFile(#PB_Any, Filename$)
    If file
      nTotalBytes = Lof(file)
      If Offset < nTotalBytes
        FileSeek(file, Offset)
        If Length = 0
          Length = nTotalBytes
        EndIf
        If Offset + Length > nTotalBytes
          Length = nTotalBytes - Offset
        EndIf
        nTotalBytes = Length
        
        SHA3_init(@ctx, Mode)
        
        While Length > 0
          nBytes = ReadData(file, @DataChunk(), #gnChunkSize)
          If nBytes
            If nBytes > Length
              nBytes = Length
            EndIf
            SHA3_update(@ctx, @DataChunk(), nBytes)
            Length - nBytes
            nTotalBytesRead + nBytes
            If ProgressBarGadgetNr >= 0 And IsGadget(ProgressBarGadgetNr)
              SetGadgetState(ProgressBarGadgetNr, 100 * nTotalBytesRead / nTotalBytes)
            EndIf
          Else
            error = #True
            Break
          EndIf
        Wend
        
        If error <> #True
          hash = SHA3_final(@ctx)
        EndIf
        
      EndIf
      CloseFile(file)
    EndIf
    
    ProcedureReturn hash
  EndProcedure
  
  Procedure$ SHA3Fingerprint(*Buffer, Size, Mode=224)             
    Define ctx.SHA3Context
    SHA3_init(@ctx, Mode)
    SHA3_update(@ctx, *Buffer, Size)
    ProcedureReturn SHA3_final(@ctx)   
  EndProcedure
  
EndModule

Kukulkan · Post by **Kukulkan** » Tue May 12, 2015 1:52 pm

Really nice, thank you!

wilbert · Post by **wilbert** » Tue May 12, 2015 5:44 pm

Kukulkan wrote:Really nice, thank you!

Thanks

I updated the post above with a small change.

Code: Select all

Debug SHA3::Fingerprint(@"", 0, 256)

will output the hash from the Draft FIPS202 specification.

Code: Select all

Debug SHA3::Fingerprint(@"", 0, 256 | SHA3::#ModeKeccak)

will output the original Keccak hash.

It was little effort to support both versions this way.

I'm thinking of adding Shake128 and Shake256 as well.
Those are also in the draft specification and are functions with a variable digest size.
You could get a 32 bit hash but also a 4096 bit hash using those functions.
The only thing is that I don't know how to add them without breaking backward compatibility since an extra parameter would be needed (digest size).

wilbert · Post by **wilbert** » Fri Aug 07, 2015 12:21 pm

As of august 5, SHA3 is finalized and finally a standard

http://keccak.noekeon.org/fips202final.html
Now the code in this thread can be verified against the test vectors of the finalized version.

Test vectors of finalized version :
https://github.com/gvanas/KeccakCodePac ... A3-224.txt
https://github.com/gvanas/KeccakCodePac ... A3-256.txt
https://github.com/gvanas/KeccakCodePac ... A3-384.txt
https://github.com/gvanas/KeccakCodePac ... A3-512.txt
https://github.com/gvanas/KeccakCodePac ... AKE128.txt
https://github.com/gvanas/KeccakCodePac ... AKE256.txt

Etayson · Post by **Etayson** » Sun Jun 05, 2016 5:07 pm

wilbert wrote:
Kukulkan wrote:Really nice, thank you!
Thanks

I updated the post above with a small change.
Code: Select all
Debug SHA3::Fingerprint(@"", 0, 256)
will output the hash from the Draft FIPS202 specification.
Code: Select all
Debug SHA3::Fingerprint(@"", 0, 256 | SHA3::#ModeKeccak)
will output the original Keccak hash.

It was little effort to support both versions this way.

I'm thinking of adding Shake128 and Shake256 as well.
Those are also in the draft specification and are functions with a variable digest size.
You could get a 32 bit hash but also a 4096 bit hash using those functions.
The only thing is that I don't know how to add them without breaking backward compatibility since an extra parameter would be needed (digest size).

hi, when i use:
IncludeFile "SHA3_x64.pbi"
Debug SHA3::Fingerprint(@"testing", 0, 256 | SHA3::#ModeKeccak)

I got error: Module item'Fingerprint' not found

Why i get this error&

infratec · Post by **infratec** » Sun Jun 05, 2016 5:21 pm

That's right.

You have to rename the procedures and the declares from

SHA3Fingerprint

to

Fingerprint

Bernd

walbus · Post by **walbus** » Sun Jun 05, 2016 7:37 pm

Wilberts codes are "Arts of work" !

Wilbert is NOT a babbler, you have a problem and ask friendly, wilbert helps !

For the SHA3 Module,
rename simple inside the module with the PB rename function, as sample so :

SHA3::Fingerprint_(*Buffer, Size, 256)

RichAlgeni · Post by **RichAlgeni** » Sun Jun 05, 2016 8:14 pm

walbus wrote:Wilberts codes are "Arts of work" !

Wilbert is NOT a babbler, you have a problem and ask friendly, wilbert helps !

For the SHA3 Module,
rename simple inside the module with the PB rename function, as sample so :

SHA3::Fingerprint_(*Buffer, Size, 256)

Here, here!!

Etayson · Post by **Etayson** » Sun Jun 26, 2016 5:25 pm

Hello all . I really need help with the module sha-3 kessak. I'm writing a program for mining Ethereum . But Sha-3 module provides the result is not as it should be.
Can someone suggest how to fix the module to the results were the same ?
Here are quotes from ethash doc (https://github.com/ethereum/wiki/wiki/Ethash)

A note regarding "SHA3" hashes described in this specification
Ethereum's development coincided with the development of the SHA3 standard, and the standards process made a late change in the padding of the finalized hash algorithm, so that Ethereum's "sha3_256" and "sha3_512" hashes are not standard sha3 hashes, but a variant often referred to as "Keccak-256" and "Keccak-512" in other contexts. See discussion, e.g. here, here, or here.

Please keep that in mind as "sha3" hashes are referred to in the description of the algorithm below.

And here is a link to sha-3 kessak used in Ethash: https://github.com/ethereum/ethash/blob ... ash/sha3.c

PureBasic Forums - English

SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)

Re: SHA3 Module (Keccak Module)