AES streaming file encoder/decoder

netmaestro · Post by **netmaestro** » Mon Aug 02, 2010 6:20 pm

There are a few wrinkles associated with streaming AES for files, and I don't think there's a sample on the forums. So here's one, it's tested successfully on files of all types from 5 bytes on up to 1gb. CRC32 File Fingerprint of the decrypted file matches that of the original in all cases. The gui isn't bulletproof, you'd want to add an Abort button to the window and put appropriate code in the threads for a graceful exit, but it shows the method:

Code: Select all

; Demo: Streaming AES file encode/decode
; netmaestro August 2010
; Purebasic 4.51

Declare Encode(void)
Declare Decode(void)

OpenWindow(0,0,0,200,120,"Cryptor!",#PB_Window_SystemMenu|#PB_Window_ScreenCentered)
ButtonGadget(0, 50,20,100,20,"Encrypt a file")
ButtonGadget(1, 50,50,100,20,"Decrypt a file")
CreateStatusBar(0, WindowID(0))
AddStatusBarField(200)
StatusBarProgress(0,0,0,#PB_ProgressBar_Smooth,0,100)

Repeat
  EventID = WaitWindowEvent()
  Select EventID
    Case #PB_Event_Gadget
      Select EventGadget()
        Case 0
          CreateThread(@Encode(), 0)
          
        Case 1
          CreateThread(@Decode(), 0)
      EndSelect
  EndSelect
Until EventID = #PB_Event_CloseWindow

End


Procedure Encode(void)
  
  chunksize = 4096
  rawfilename$ = OpenFileRequester("Please choose a file to encrypt:","","",0)
  encfilename$ = rawfilename$+".enc"
  
  If OpenFile(0, rawfilename$)
    If CreateFile(1, encfilename$)
      DisableGadget(0,1)
      DisableGadget(1,1)
      
      length.q = Lof(0)
      numparts = length/chunksize
      lastchunksize = length%chunksize
      If lastchunksize
        numparts+1
        lastchunk = numparts
      Else
        lastchunk = 0
      EndIf
      
      *raw     = AllocateMemory(chunksize) 
      *secure  = AllocateMemory(chunksize) 
      
      StartAESCipher(0, ?key, 256, ?iVector, #PB_Cipher_CBC|#PB_Cipher_Encode)
      
      For i=1 To numparts
        If i=lastchunk
          FillMemory(*raw, chunksize, 0, #PB_Byte)
          ReadData(0, *raw, lastchunksize)
        Else
          ReadData(0, *raw, chunksize)
        EndIf
        AddCipherBuffer(0, *raw, *secure, chunksize)
        WriteData(1, *secure, chunksize)
        prog.d = i/numparts*100
        StatusBarProgress(0,0, Int(prog))
        Delay(1)
      Next
      If lastchunksize
        WriteLong(1, lastchunksize)
      Else
        WriteLong(1, chunksize)
      EndIf
      
      FinishCipher(0) 
      CloseFile(0)
      CloseFile(1)
      
      FreeMemory(*raw)
      FreeMemory(*secure)
      
      MessageRequester("Finished", "Encrypted file "+encfilename$+" saved",#MB_ICONINFORMATION)
      StatusBarProgress(0,0,0)
      
      DisableGadget(0,0)
      DisableGadget(1,0)
      
    EndIf
  EndIf
EndProcedure

Procedure Decode(void)
  
  chunksize = 4096
  
  encfilename$ = OpenFileRequester("Please choose a file to decrypt:","","",0)
  rawfilename$ = SaveFileRequester("Please choose a new name for the decrypted file","","",0)
  
  If OpenFile(0, encfilename$)
    FileSeek(0, Lof(0)-4)
    lastchunksize = ReadLong(0)
    FileSeek(0, 0)
    If CreateFile(1, rawfilename$)
      DisableGadget(0,1)
      DisableGadget(1,1)
      
      length.q = Lof(0)
      numparts = length/chunksize
      
      *raw     = AllocateMemory(chunksize) 
      *secure  = AllocateMemory(chunksize) 
      
      StartAESCipher(0, ?key, 256, ?iVector, #PB_Cipher_CBC|#PB_Cipher_Decode)
      
      For i=1 To numparts
        ReadData(0, *secure, chunksize)
        AddCipherBuffer(0, *secure, *raw, chunksize)
        If i=numparts
          WriteData(1, *raw, lastchunksize)
        Else
          WriteData(1, *raw, chunksize)
        EndIf
        prog.d = i/numparts*100
        StatusBarProgress(0,0, Int(prog))
        Delay(1)
      Next
      
      FinishCipher(0) 
      CloseFile(0)
      CloseFile(1)
      
      FreeMemory(*raw)
      FreeMemory(*secure)
      
      MessageRequester("Finished", "Decrypted file "+rawfilename$+" saved",#MB_ICONINFORMATION)
      StatusBarProgress(0,0,0)
      
      DisableGadget(0,0)
      DisableGadget(1,0)
    EndIf
  EndIf
EndProcedure


DataSection
  key:
  Data.b $8C,$15,$51,$2C,$0C,$8A,$0A,$D8,$07,$E4,$21,$A2,$8E,$83,$A3,$88,$8A,$CA,$FB,$E1
  Data.b $7B,$A3,$6B,$D6,$BC,$F7,$E6,$CD,$FE,$B5,$D7,$B3
  iVector:
  Data.b $08,$0C,$96,$48,$33,$51,$35,$80,$0C,$A9,$42,$1E,$11,$E0,$83,$C7,$C4,$C6,$E1,$E4
  Data.b $2E,$40,$81,$0A,$24,$70,$00,$10,$08,$B3,$64,$21
EndDataSection

A couple of things to keep in mind:

1) For 256bit encoding, your key and initialization vector should be 32 bytes in length.

2) For 256bit encoding, you should choose a chunk size which is a multiple of 2 and at least 32 bytes.

3) All your streamed buffers must be of equal size, including the last one. This goes for both Encode and Decode.

4) When you decode a file, you need to read all buffers in at the full chunk size. However, you will only want to write the last buffer out to the target file at its original (usually less-than-chunksize) size. Otherwise your target file will not be the same as the original file you encrypted. How do you know what this size was? What I did was write it out to the end of the target file at the encryption stage, and then read it in again at the decryption stage. This way you can AddCipherBuffer at the full chunksize for the decrypt but only write out those decoded bytes that matter. The remainder of the last buffer can safely be discarded.

netmaestro · Post by **netmaestro** » Tue Aug 03, 2010 2:55 pm

Tiny change to the code today, length var is changed to quad. It allows for files larger than 2gb to be processed. So far the largest file I tested was a 4.5 gb dvd iso image. It took about 20 minutes to encrypt and another 20 to decrypt, my computer isn't that fast. But the result was without flaw. I'm considering a cooler gui and some more features for this as I think it's got potential as a handy tool.

KJ67 · Post by **KJ67** » Tue Aug 03, 2010 3:21 pm

Nice code!

I cannot get much more the 5-10% CPU-load, it seems just to be an issue of disk-access for me. Interesting thou, when I run it the code it peaks at ~ 12 threads and slowly drops down to 4 while the CPU load is pretty much even... It doesn’t say much in the manual but I guess the AES is multi threaded directly in the library.
Anyone who knows more?

Post by **Fred** » Tue Aug 17, 2010 9:00 am

For information, AES isn't threaded in the library (for now).

new2pb · Post by **new2pb** » Tue Aug 17, 2010 2:52 pm

@netmaestro

Is it possible for an encrypted chunk to be decrypted in memory rather than saving to a file? I want to see if its possible to place it into an array after decrypting it.

This was taken from your source above

StartAESCipher(0, ?key, 256, ?iVector, #PB_Cipher_CBC|#PB_Cipher_Decode)

For i=1 To numparts
ReadData(0, *secure, chunksize)
AddCipherBuffer(0, *secure, *raw, chunksize)
If i=numparts
WriteData(1, *raw, lastchunksize)
Else
WriteData(1, *raw, chunksize)
EndIf
prog.d = i/numparts*100
StatusBarProgress(0,0, Int(prog))
Delay(1)
Next

netmaestro · Post by **netmaestro** » Tue Aug 17, 2010 2:56 pm

Code: Select all

AddCipherBuffer(0, *secure, *raw, chunksize)

At this point the memory block at *raw is decrypted. I'm just writing it out to the fille. You could save the chunks to an array if you wanted to I guess. But if you were going to do that you might as well use AESDecoder() instead which doesn't do it in chunks.

new2pb · Post by **new2pb** » Tue Aug 17, 2010 3:10 pm

netmaestro wrote:
Code: Select all
AddCipherBuffer(0, *secure, *raw, chunksize)
At this point the memory block at *raw is decrypted. I'm just writing it out to the fille. You could save the chunks to an array if you wanted to I guess. But if you were going to do that you might as well use AESDecoder() instead which doesn't do it in chunks.

Thanks for your fast response,

I not too sure of the differences between addcipherbuffer and aesdecoder.

netmaestro · Post by **netmaestro** » Tue Aug 17, 2010 3:17 pm

AESDecoder() / AESEncoder() work on one memory block of any size in one pass. AddCipherBuffer() works one chunk at a time, which allows you to stream in and out. So if you have a 7gb file, that won't fit in memory and you couldn't encrypt/decrypt it without the streaming option. For anything up to say 100mb or so you should be fine with AESEncoder(), it's simpler.

new2pb · Post by **new2pb** » Tue Aug 17, 2010 3:23 pm

Thanks, I will try again. Using aesencoder and aesdecoder.

new2pb · Post by **new2pb** » Tue Aug 17, 2010 5:35 pm

data.prefs

[Directory]
path1=C:\xampplite\htdocs
path2=D:\xampplite\htdocs
[Settings]
directoryupdates= Yes

Demo.pb

Global emulateFile$,arraysizeI.i

Procedure.i startDataI(fileToCount$)
arraysizeI.i=0
If FileSize(fileToCount$)=-1
ProcedureReturn #False
Else
If ReadFile(1, fileToCount$)
While Eof(1) =0
ReadString(1)
arraysizeI.i=arraysizeI.i+1
Wend
CloseFile(1)
ProcedureReturn arraysizeI.i
Else
ProcedureReturn arraysizeI.i
EndIf
EndIf
EndProcedure

Procedure.s startDataII(Array A.s(1))
Define loopnum.i=0
Define xchunk= 4096
If FileSize(emulateFile$)=-1
ProcedureReturn Str(#False)
Else
If ReadFile(1, emulateFile$)
While Eof(1) = 0
*CipheredString= AllocateMemory(xchunk)
Define text.s=ReadString(1)
AESEncoder(@text, *CipheredString, Len(text.s), ?Key, 128, ?InitializationVector)
A(loopnum) = PeekS(*CipheredString)
loopnum.i=loopnum.i + 1
FreeMemory(*CipheredString)
Wend
CloseFile(1)
ProcedureReturn Str(#True)
Else
ProcedureReturn Str(#False)
EndIf
EndIf
EndProcedure

Procedure decodeDataII(Array B.s(1),filename$)
Define loopnum.i=0
Define xchunkz= 4096
If FileSize(filename$)=-1
ProcedureReturn #False
Else
If ReadFile(1, filename$)
While Eof(1) = 0
*DecipheredString = AllocateMemory(xchunkz) ; null terminating character (ASCII mode)
Define text.s=ReadString(1)
Define lenght=Len(ReadString(1))
AESDecoder(@text, *DecipheredString, lenght, ?Key, 128, ?InitializationVector)
B(loopnum) = PeekS(*DecipheredString)
loopnum.i=loopnum + 1
FreeMemory(*DecipheredString)
Wend
CloseFile(1)
ProcedureReturn #True
Else
ProcedureReturn #False
EndIf
EndIf
EndProcedure

Procedure startDataIII(Array A.s(1),sizeOfArray.i,fileToCreate$)
Define i.i
If CreateFile(0, fileToCreate$)
For i = 0 To sizeOfArray.i
FileSeek(0, Lof(0))
WriteStringN(0, A(i))
Next i
CloseFile(0)
EndIf
EndProcedure

Procedure decodeDataIII(Array B.s(1),sizeOfArray.i,fileToCreate$)
Define i.i
If CreateFile(0, fileToCreate$)
For i = 0 To sizeOfArray.i
FileSeek(0, Lof(0))
WriteStringN(0, B(i))
Next i
CloseFile(0)
EndIf
EndProcedure

DataSection
Key:
Data.b $43,$8B,$1E,$9F,$EC,$2B,$4E,$48,$A0,$02,$E1,$C8,$D8,$7B,$3D,$A1

InitializationVector:
Data.b $3d, $af, $ba, $42, $9d, $9e, $b4, $30, $b4, $22, $da, $80, $2c, $9f, $ac, $41
EndDataSection

Define Pattern$ = "Text (*.txt)|*.txt;*.bat|PureBasic (*.pb)|*.pb|All files (*.*)|*.*"
Define StandardFile$ = GetPathPart(ProgramFilename())
Define Pattern = 0
emulateFile$ = OpenFileRequester("Please choose a file to encrypt:",StandardFile$, Pattern$, Pattern)
If emulateFile$
startDataI(emulateFile$)
arraysizeI.i= arraysizeI.i-1
Dim A.s(100)
startDataII(A())
Define File$ = SaveFileRequester("Save Encode File", StandardFile$, Pattern$, Pattern)
If File$
startDataIII(A(),arraysizeI.i,File$)
Else
MessageRequester("Information", "The requester was canceled.", 0)
EndIf
startDataI(File$)
arraysizeI.i= arraysizeI.i-1
Dim B.s(100)
decodeDataII(B(),File$)
Define File$ = SaveFileRequester("Save Decode File", StandardFile$, Pattern$, Pattern)
If File$
decodeDataIII(B(),arraysizeI.i,File$)
Else
MessageRequester("Information", "The requester was canceled.", 0)
EndIf
Else
MessageRequester("Information", "The requester was canceled.", 0)
EndIf

Hi netmaestro,

This is my little demo to try out to see the output and also trying out the array but some how.
decodeDataII(Array B.s(1),filename$) <--- this procedure is not working... I cannot decode my encoded portion

Been trying hard to understand how to manipulate.

netmaestro · Post by **netmaestro** » Tue Aug 17, 2010 6:04 pm

I partially fixed it for you, it will work now but with errors. See the comment re: PeekS() for the reason. You must use something other than PeekS() to store the encrypted string.

Code: Select all

Global emulateFile$,arraysizeI.i

Procedure.i startDataI(fileToCount$)
  arraysizeI.i=0
  If FileSize(fileToCount$)=-1
    ProcedureReturn #False
  Else
    If ReadFile(1, fileToCount$)
      While Eof(1) =0
        ReadString(1)
        arraysizeI.i=arraysizeI.i+1
      Wend
      CloseFile(1)
      ProcedureReturn arraysizeI.i
    Else
      ProcedureReturn arraysizeI.i
    EndIf
  EndIf
EndProcedure

Procedure.s startDataII(Array A.s(1))
  Define loopnum.i=0
  Define xchunk= 4096
  If FileSize(emulateFile$)=-1
    ProcedureReturn Str(#False)
  Else
    If ReadFile(1, emulateFile$)
      While Eof(1) = 0 
        *CipheredString= AllocateMemory(xchunk)
        Define text.s=ReadString(1)
        If Len(text)>0
          AESEncoder(@text, *CipheredString, Len(text.s), ?Key, 128, ?InitializationVector)
          EndIf
          A(loopnum) = PeekS(*CipheredString) ; < This is a mistake, the ciphered string will contain null bytes and your
          loopnum.i=loopnum.i + 1             ;    string in A(loopnum) will be truncated
        FreeMemory(*CipheredString)
      Wend
      CloseFile(1)
      ProcedureReturn Str(#True)
    Else
      ProcedureReturn Str(#False)
    EndIf
  EndIf
EndProcedure

Procedure decodeDataII(Array B.s(1),filename$)
  Define loopnum.i=0
  Define xchunkz= 4096
  If FileSize(filename$)=-1
    ProcedureReturn #False
  Else
    If ReadFile(1, filename$)
      While Eof(1) = 0
        *DecipheredString = AllocateMemory(xchunkz) ; null terminating character (ASCII mode)
        Define text.s=ReadString(1)
        Define lenght = Len(text)             ;lenght=Len(ReadString(1)) <-------------- This was a mistake, you're reading the next string
        If lenght
          AESDecoder(@text, *DecipheredString, lenght, ?Key, 128, ?InitializationVector)
        EndIf
        B(loopnum) = PeekS(*DecipheredString) 
        loopnum.i=loopnum + 1
        FreeMemory(*DecipheredString)
      Wend
      CloseFile(1)
      ProcedureReturn #True
    Else
      ProcedureReturn #False
    EndIf
  EndIf
EndProcedure

Procedure startDataIII(Array A.s(1),sizeOfArray.i,fileToCreate$)
  Define i.i
  If CreateFile(0, fileToCreate$)
    For i = 0 To sizeOfArray.i
      FileSeek(0, Lof(0))
      WriteStringN(0, A(i))
    Next i
    CloseFile(0)
  EndIf
EndProcedure

Procedure decodeDataIII(Array B.s(1),sizeOfArray.i,fileToCreate$)
  Define i.i
  If CreateFile(0, fileToCreate$)
    For i = 0 To sizeOfArray.i
      FileSeek(0, Lof(0))
      WriteStringN(0, B(i))
    Next i
    CloseFile(0)
  EndIf
EndProcedure

DataSection
  Key:
  Data.b $43,$8B,$1E,$9F,$EC,$2B,$4E,$48,$A0,$02,$E1,$C8,$D8,$7B,$3D,$A1
  
  InitializationVector:
  Data.b $3d, $af, $ba, $42, $9d, $9e, $b4, $30, $b4, $22, $da, $80, $2c, $9f, $ac, $41
EndDataSection

Define Pattern$ = "Text (*.txt)|*.txt;*.bat|PureBasic (*.pb)|*.pb|All files (*.*)|*.*"
Define StandardFile$ = GetPathPart(ProgramFilename())
Define Pattern = 0
emulateFile$ = OpenFileRequester("Please choose a file to encrypt:",StandardFile$, Pattern$, Pattern)
If emulateFile$
  startDataI(emulateFile$)
  arraysizeI.i= arraysizeI.i-1
  Dim A.s(100)
  startDataII(A())
  Define File$ = SaveFileRequester("Save Encode File", StandardFile$, Pattern$, Pattern)
  If File$
    startDataIII(A(),arraysizeI.i,File$)
  Else
    MessageRequester("Information", "The requester was canceled.", 0) 
  EndIf
  startDataI(File$)
  arraysizeI.i= arraysizeI.i-1
  Dim B.s(100)
  decodeDataII(B(),File$)
  Define File$ = SaveFileRequester("Save Decode File", StandardFile$, Pattern$, Pattern)
  If File$
    decodeDataIII(B(),arraysizeI.i,File$)
  Else
    MessageRequester("Information", "The requester was canceled.", 0) 
  EndIf
Else
  MessageRequester("Information", "The requester was canceled.", 0) 
EndIf

new2pb · Post by **new2pb** » Tue Aug 17, 2010 6:54 pm

Hmm, beside PeekS what others can store encrypted strings especially with null and bytes...

I guess I have try every single one to see its effects.... no idea.... currently....

anyway thanks netmaestro for the tips up... I think I shall stop from here for a while...brain cannot think at 1:53am here.... time for bed

Little John · Post by **Little John** » Wed Aug 18, 2010 2:32 pm

For storing memory content that might contain null bytes, I recommend my function PeekHexBytes.

Or convert the respective memory area e.g. with Base64Encoder(), before storing its content in a string.

Regards, Little John

new2pb · Post by **new2pb** » Wed Aug 18, 2010 2:38 pm

Little John wrote:For storing memory content that might contain null bytes, I recommend my function PeekHexBytes.
Or convert the respective memory area e.g. with Base64Encoder(), before storing its content in a string.

Regards, Little John

Hi John and netmaestro,

Thanks for the great help. In the end I use base64 together with aes

epidemicz · Post by **epidemicz** » Fri Sep 03, 2010 7:31 am

Very cool indeed

PureBasic Forums - English

AES streaming file encoder/decoder

AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder

Re: AES streaming file encoder/decoder