Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Share your advanced PureBasic knowledge/code with the community.
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Data_Tool_BF - Data Generator - 50% smaller output + AES + Compression - Modules

Post by Saki »

DATA_Tool_BF Unicode Data Generator - 50% smaller output

The tool provides a high level of protection against the manipulation or alteration of images embedded in the codes. :!:
The pictures cannot be changed or localised by strangers if one does not
know exactly what is there or what exactly is happening.

The embedded data is automatically encrypted.

There are two variants.
The first with a hard-wired automatic encryption,
which does not require a password.
However, an integer number can be set as the password.
This encryption is very difficult to explore and the encrypted data is difficult to locate.
The second variant is exactly the same, but uses an AES encryption which can be passed a password.
Encryption is always used, even if no variable or password is defined.

Also variants with LZMA compression are available

It is absolutely simple.
Anyone can create an encrypted image in Data lines and include it in their code in an executable way in less than 30 seconds.


To test the module, proceed as follows:

Select the Encoder.
Enter an image in the file requester that appears.
Your clipboard now contains the image you selected as a finished data section.
Replace the data lines in the demo part from the decoder code with the content of the clipboard.
Now your new image is embedded in the code.
Start the decoder code and your new image should now be displayed.

The Encoder can generate huge amounts of data lines blazing fast, as sample, more as 700,000 lines in one second.
The Decoder is also extremely fast.

A version which does not require the PB Data section can be found here :
viewtopic.php?f=12&t=77295&p=570371#p570371


Encoder - High-Speed - Can create more as 700,000 data lines in one second.

Code: Select all

DeclareModule Data_Tool_Encoder_BF
  EnableExplicit
  ; Sakis data Generator © - Encoder - Using StringFields_BF - Incredible fast
  ; This code is free for use, changing and enhancing
  ; character_offset - Base character offset - As sample 30, 160, 230 - Try what ever you want
  ; Seed - set as a integer - This works as password for the integrated data crypter
  Declare Data_Encoder_BF(path$="", length_data_line=100, character_offset=230, seed=0)
EndDeclareModule

Module Data_Tool_Encoder_BF
  ; StringField_Tool_BF - By Saki - Unicode - This code is free for using and enhancing
  Global NewList index(), NewList indexes.s(), empty_fields.q, start_index.q=1, end_index.q=-1
  AddElement(index()) : AddElement(indexes())
  Procedure StringFields_BF(string$, separator$, mode=1, ignore_empty_fields=1)
    Select mode
      Case 0 : Protected add_first=1 : Case 2 : Protected add_last=1 : Case 3 : add_first=1 : add_last=1
    EndSelect
    Protected skip_first : If start_index<1 : start_index=1 : EndIf
    Protected i, ii, iii, iiii, pos_1, pos_2, length_result, comp, count_index, amount_indexes
    Protected len_separator=StringByteLength(separator$), skip_last, *string=@string$
    Protected *separator=@separator$, *pointer.word, byte_pos_last, result$ 
    If end_index=-1 : end_index=1e18 : EndIf : end_index-1
    If start_index>end_index : start_index=end_index+1 : EndIf : If Not PeekW(*string) : ProcedureReturn 0 : EndIf
    ClearList(index()) : AddElement(index()) : ClearList(indexes()) : AddElement(indexes())
    If comp=CompareMemory(*string, *separator, len_separator) : end_index+1 : count_index+1 : skip_first=1 : EndIf
    If add_first
      If skip_first And start_index : start_index-2 : end_index-1 : EndIf
    Else
      If skip_first And start_index : start_index-1 : EndIf 
    EndIf
    i=-2
    Repeat
      i+2 : comp=CompareMemory(*string+i, *separator, len_separator)
      If comp
        iii=i : count_index+1 : ii+1 : i+len_separator-2 : amount_indexes+1 : AddElement(index()) : index()=i+2
      EndIf 
      *pointer=*string+i+len_separator
    Until count_index>end_index Or Not *pointer\w
    iiii=i : byte_pos_last=iii+len_separator
    If comp=CompareMemory(*string+i-len_separator-1, *separator, len_separator) : skip_last=1 : EndIf
    If end_index>count_index : end_index=count_index : EndIf : amount_indexes=ii : i=skip_first
    If start_index>amount_indexes : start_index=amount_indexes-1 : EndIf : i+start_index+skip_first+skip_last
    If skip_first : i-1 : EndIf : If skip_last : i-1 : EndIf
    If amount_indexes
      Repeat 
        If ListSize(index())>i : SelectElement(index(), i) : pos_1=index() : EndIf
        If ListSize(index())>i+1 : SelectElement(index(), i+1) : pos_2=index() : EndIf
        length_result=(pos_2-pos_1-len_separator)>>1
        If pos_2-pos_1>0
          If length_result>0
            result$=PeekS(*string+pos_1, length_result) : AddElement(indexes()) : indexes()=result$ 
          Else 
            empty_fields+1
            If ignore_empty_fields : result$=#Null$ : Else : result$="" : AddElement(indexes()) : EndIf         
          EndIf
        EndIf
        i+1
      Until i>end_index Or i=amount_indexes
      If add_last And skip_last
        result$=PeekS(*string+byte_pos_last, (iiii-byte_pos_last+len_separator)>>1)
        AddElement(indexes()) : indexes()=result$ 
      EndIf
    EndIf
    ProcedureReturn 1
  EndProcedure
  
  Procedure.s GetStringFields_BF(field)
    If field <1 : ProcedureReturn "" : EndIf
    SelectElement(indexes(), field) : If field<ListSize(indexes()) : ProcedureReturn indexes() : EndIf
  EndProcedure
  
  Procedure FreeAllStringFields_BF()
    ClearList(index()) : AddElement(index()) : ClearList(indexes()) : AddElement(indexes())
    empty_fields.q=0 : start_index.q=1 : end_index.q=-1 : ProcedureReturn 1
  EndProcedure
  
  Procedure Data_Encoder_BF(path$="", length_data_line=100, character_offset=230, seed=0)
    ; Sakis data Generator - This code put the generated data in your Clipboard - You see nothing
    Protected i, ii, iii, breaks, *buffer, len_buffer, file, lof_file, len_source.q, padding=3
    Protected string$, new_string$, temp_string$, last$, length$
    Macro Create_length_string
      length$=LSet(Chr(character_offset<<8), 6, Chr(character_offset<<8))
      ii=0 : For i=0 To 5 : PokeA(@length$+ii, PeekA(@len_source+i)) : ii+2 : Next i
    EndMacro
    If path$="" : path$=OpenFileRequester("Select source", "", "", 0) : If path$="" : ProcedureReturn 0 : EndIf : EndIf
    If FileSize(path$)<1 : ProcedureReturn 0 : EndIf
    file=ReadFile(#PB_Any, path$) : If Not file : ProcedureReturn 0 : EndIf
    lof_file=Lof(file) : *buffer=AllocateMemory(lof_file)
    If Not ReadData(file, *buffer, lof_file) : CloseFile(file) : ProcedureReturn 0 : EndIf : CloseFile(file)
    len_buffer=lof_file+padding : len_source.q=len_buffer-3 : RandomSeed(seed)
    breaks=len_buffer/length_data_line : breaks+Bool(len_buffer%length_data_line) : string$=Space(len_buffer+breaks-1)
    For i=0 To len_buffer-1 : ii+1 : iii+1
      PokeA(@string$+i+ii, character_offset) : PokeA(@string$+i+ii-1, PeekA(*buffer+i)+Random(254, 1))
      If Not iii%length_data_line : ii+1 : PokeW(@string$+i+ii, 10) : ii+1 : EndIf
    Next 
    If breaks<2
      create_length_string : SetClipboardText("Data.s "+Chr(34)+string$+Chr(34)+","+Chr(34)+length$+Chr(34))
      ProcedureReturn 1
    EndIf
    iii=(length_data_line+10) : new_string$=Space(breaks*iii) : ii=0 : iii*2
    StringFields_BF(string$, #LF$, 3)
    For i=1 To breaks
      temp_string$="Data.s "+Chr(34)+GetStringFields_BF(i)+Chr(34)+#LF$
      CopyMemory(@temp_string$, @new_string$+ii, iii) : ii+iii 
    Next
    new_string$+"Data.s "+Chr(34) : Create_length_string : new_string$+length$+Chr(34)
    SetClipboardText(new_string$) : FreeAllStringFields_BF()
    ProcedureReturn 1
  EndProcedure
EndModule
UseModule Data_Tool_Encoder_BF

; #################### Encode - You found the data in your Clipboard ########################
CompilerIf #PB_Compiler_IsMainFile
  Define len_line=100
  ; Data_Encoder_BF() ; Preset
  Data_Encoder_BF("", len_line)
  MessageRequester("Encoder", "Encode finished"+#LF$+#LF$+ "The data are now in your Clipboard")
CompilerEndIf

Decoder :

Code: Select all

DeclareModule Data_Tool_Decoder_BF
  EnableExplicit
  ; Sakis data Generator - © Decoder
  ; This code is free for use, changing and enhancing
  ; character_offset - Base character offset - As sample 30, 160, 230 - Try what ever you want
  ; Seed - set as a integer - This works as password for the integrated data crypter
  Declare Data_Decoder_BF(*data_start_pointer, *data_end_pointer, seed=0)
EndDeclareModule

Module Data_Tool_Decoder_BF
   ; You can remove what you not want
  UsePNGImageDecoder() : UseJPEGImageDecoder() : UseGIFImageDecoder() : UseTIFFImageDecoder()
  
  Procedure Data_Decoder_BF(*data_start_pointer, *data_end_pointer, seed=0)
    ; Sakis data Generator - This function give back a pointer to a allocated memory with your data
    Protected i, ii, breaks, *buffer, *catch_buffer, len_buffer, len_line
    Protected len_data.q, data_size, *catch_buffer_new : len_line=Len(PeekS(*data_start_pointer))
    data_size=*data_end_pointer-*data_start_pointer : len_buffer=data_size/2 : RandomSeed(seed)
    *buffer=AllocateMemory(data_size) : CopyMemory(*data_start_pointer, *buffer, data_size)
     Repeat : If Not PeekW(*buffer+i) : PokeW(*buffer+i, 2560) : breaks+1 : EndIf : i+2 : Until i>data_size
    *catch_buffer=AllocateMemory(data_size/2-breaks)
    For i=0 To len_buffer-breaks-1 : If Not i%len_line And i : ii+2 : EndIf
      PokeA(*catch_buffer+i, PeekA(*buffer+ii)-Random(254, 1)) : ii+2
    Next
    ii=14 : For i=0 To 5 : PokeA(@len_data+i, PeekA(*data_end_pointer-ii)) : ii-2 : Next i
    *catch_buffer_new=AllocateMemory(len_data)
    CopyMemory(*catch_buffer, *catch_buffer_new, MemorySize(*catch_buffer_new))
    FreeMemory(*catch_buffer) : FreeMemory(*buffer)
    ProcedureReturn *catch_buffer_new
  EndProcedure 
EndModule
UseModule Data_Tool_Decoder_BF

; #################### Decoder Demo part ########################
CompilerIf #PB_Compiler_IsMainFile
  EnableExplicit
  
  Define *catch_buffer=Data_Decoder_BF(?image_1_start, ?image_1_end)
  
  Define image_ID=CatchImage(#PB_Any, *catch_buffer)
  FreeMemory(*catch_buffer)
  
  Define window_ID=OpenWindow(#PB_Any, 0, 0, 600, 400, "", #PB_Window_SystemMenu|#PB_Window_ScreenCentered)
  If IsImage(image_ID)
    Define image_gadget_ID=ImageGadget(#PB_Any,
                                       WindowWidth(window_ID)/2-ImageWidth(image_ID)/2,
                                       WindowHeight(window_ID)/2-ImageHeight(image_ID)/2,
                                       0,
                                       0,
                                       ImageID(image_ID))
  Else
    CloseWindow(window_ID)
    MessageRequester("Hint", "Embedded image not usable")
    End
  EndIf
  
  Repeat : Until WaitWindowEvent()=#PB_Event_CloseWindow
  
  DataSection
    image_1_start:
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    image_1_end:
  EndDataSection
CompilerEndIf

Encoder with AES encryption and crypt randomized IV

Code: Select all

DeclareModule Data_Tool_Encoder_AES_BF
  EnableExplicit
  ; Sakis data Generator © - Encoder - Using StringFields_BF - Incredible fast
  ; Variant with integrated AES encryption and automatically crypt randomized IV
  ; This code is free for use, changing and enhancing
  ; character_offset - Base character offset - As sample 30, 160, 230 - Try what ever you want
  ; Seed - set as a integer - This works as password for the integrated data crypter
  Declare Data_Encoder_AES_BF(path$="", length_data_line=100, character_offset=230, password$="")
EndDeclareModule

Module Data_Tool_Encoder_AES_BF
  UseMD5Fingerprint()
  ; StringField_Tool_BF - By Saki - Unicode - This code is free for using and enhancing
  Global NewList index(), NewList indexes.s(), empty_fields.q, start_index.q=1, end_index.q=-1
  AddElement(index()) : AddElement(indexes())
  Procedure StringFields_BF(string$, separator$, mode=1, ignore_empty_fields=1)
    Select mode
      Case 0 : Protected add_first=1 : Case 2 : Protected add_last=1 : Case 3 : add_first=1 : add_last=1
    EndSelect
    Protected skip_first : If start_index<1 : start_index=1 : EndIf
    Protected i, ii, iii, iiii, pos_1, pos_2, length_result, comp, count_index, amount_indexes
    Protected len_separator=StringByteLength(separator$), skip_last, *string=@string$
    Protected *separator=@separator$, *pointer.word, byte_pos_last, result$ 
    If end_index=-1 : end_index=1e18 : EndIf : end_index-1
    If start_index>end_index : start_index=end_index+1 : EndIf : If Not PeekW(*string) : ProcedureReturn 0 : EndIf
    ClearList(index()) : AddElement(index()) : ClearList(indexes()) : AddElement(indexes())
    If comp=CompareMemory(*string, *separator, len_separator) : end_index+1 : count_index+1 : skip_first=1 : EndIf
    If add_first
      If skip_first And start_index : start_index-2 : end_index-1 : EndIf
    Else
      If skip_first And start_index : start_index-1 : EndIf 
    EndIf
    i=-2
    Repeat
      i+2 : comp=CompareMemory(*string+i, *separator, len_separator)
      If comp
        iii=i : count_index+1 : ii+1 : i+len_separator-2 : amount_indexes+1 : AddElement(index()) : index()=i+2
      EndIf 
      *pointer=*string+i+len_separator
    Until count_index>end_index Or Not *pointer\w
    iiii=i : byte_pos_last=iii+len_separator
    If comp=CompareMemory(*string+i-len_separator-1, *separator, len_separator) : skip_last=1 : EndIf
    If end_index>count_index : end_index=count_index : EndIf : amount_indexes=ii : i=skip_first
    If start_index>amount_indexes : start_index=amount_indexes-1 : EndIf : i+start_index+skip_first+skip_last
    If skip_first : i-1 : EndIf : If skip_last : i-1 : EndIf
    If amount_indexes
      Repeat 
        If ListSize(index())>i : SelectElement(index(), i) : pos_1=index() : EndIf
        If ListSize(index())>i+1 : SelectElement(index(), i+1) : pos_2=index() : EndIf
        length_result=(pos_2-pos_1-len_separator)>>1
        If pos_2-pos_1>0
          If length_result>0
            result$=PeekS(*string+pos_1, length_result) : AddElement(indexes()) : indexes()=result$ 
          Else 
            empty_fields+1
            If ignore_empty_fields : result$=#Null$ : Else : result$="" : AddElement(indexes()) : EndIf         
          EndIf
        EndIf
        i+1
      Until i>end_index Or i=amount_indexes
      If add_last And skip_last
        result$=PeekS(*string+byte_pos_last, (iiii-byte_pos_last+len_separator)>>1)
        AddElement(indexes()) : indexes()=result$ 
      EndIf
    EndIf
    ProcedureReturn 1
  EndProcedure
  
  Procedure.s GetStringFields_BF(field)
    If field <1 : ProcedureReturn "" : EndIf
    SelectElement(indexes(), field) : If field<ListSize(indexes()) : ProcedureReturn indexes() : EndIf
  EndProcedure
  
  Procedure FreeAllStringFields_BF()
    ClearList(index()) : AddElement(index()) : ClearList(indexes()) : AddElement(indexes())
    empty_fields.q=0 : start_index.q=1 : end_index.q=-1 : ProcedureReturn 1
  EndProcedure
  
  Procedure Data_Encoder_AES_BF(path$="", length_data_line=100, character_offset=230, password$="")
    ; Sakis data Generator - This code put the generated data in your Clipboard - You see nothing
    Protected i, ii, iii, i1, depth, breaks, *buffer, len_buffer, file, lof_file, len_source.q, padding=3
    Protected string$, new_string$, temp_string$, last$, length$, iv$
    Protected fixed$=StringFingerprint(password$+"%$(s4DäÖÄö", #PB_Cipher_MD5) : Dim register.q(3)
    Repeat ; 16 Bytes
      PokeA(@register(0)+i, Val("$"+PeekS(@fixed$+ii, 2))) : ii+SizeOf(character)<<1 : i+1 ; Create a key
    Until ii=StringByteLength(fixed$) : i=0 : ii=0
    If OpenCryptRandom() : CryptRandomData(@register(2), 16) : Else : RandomData(@register(2), 16) : EndIf
    register(0)!register(2) : register(1)!register(3) 
    Macro Create_length_string
      length$=LSet(Chr(character_offset<<8), 6, Chr(character_offset<<8))
      ii=0 : For i=0 To 5 : PokeA(@length$+ii, PeekA(@len_source+i)) : ii+2 : Next i
    EndMacro
    Macro Create_string
      iv$=LSet(Chr(character_offset<<8), 16, Chr(character_offset<<8))
      ii=0 : For i=0 To 15 : PokeA(@iv$+ii, PeekA(@register(2)+i)) : ii+2 : Next i
    EndMacro
    If path$="" : path$=OpenFileRequester("Select source", "", "", 0) : If path$="" : ProcedureReturn 0 : EndIf : EndIf
    If FileSize(path$)<1 : ProcedureReturn 0 : EndIf
    file=ReadFile(#PB_Any, path$) : If Not file : ProcedureReturn 0 : EndIf
    lof_file=Lof(file) : *buffer=AllocateMemory(lof_file)
    If Not ReadData(file, *buffer, lof_file) : CloseFile(file) : ProcedureReturn 0 : EndIf : CloseFile(file)
    len_buffer=lof_file+padding : len_source.q=len_buffer-3
    breaks=len_buffer/length_data_line : breaks+Bool(len_buffer%length_data_line) : string$=Space(len_buffer+breaks-1)
    For i=0 To len_buffer-1 : ii+1 : iii+1
      Repeat
        If Not i1 : AESEncoder(@register(0), @register(0), 16, @register(0), 128, 0, #PB_Cipher_ECB) : EndIf
        depth=PeekA(@register(0)+i1) : i1+1 : If i1>15 : i1=0 : EndIf
      Until depth
      PokeA(@string$+i+ii, character_offset) : PokeA(@string$+i+ii-1, PeekA(*buffer+i)+depth)
      If Not iii%length_data_line : ii+1 : PokeW(@string$+i+ii, 10) : ii+1 : EndIf
    Next 
    If breaks<2
      Create_length_string : Create_string 
      SetClipboardText("Data.s "+Chr(34)+string$+Chr(34)+","+Chr(34)+iv$+length$+Chr(34))
      ProcedureReturn 1
    EndIf
    iii=(length_data_line+10) : new_string$=Space(breaks*iii) : ii=0 : iii*2
    StringFields_BF(string$, #LF$, 3)
    For i=1 To breaks
      temp_string$="Data.s "+Chr(34)+GetStringFields_BF(i)+Chr(34)+#LF$
      CopyMemory(@temp_string$, @new_string$+ii, iii) : ii+iii 
    Next
    new_string$+"Data.s "+Chr(34) : Create_length_string : Create_string : new_string$+iv$+length$+Chr(34)
    SetClipboardText(new_string$) : FreeAllStringFields_BF()
    ProcedureReturn 1
  EndProcedure
EndModule
UseModule Data_Tool_Encoder_AES_BF

; #################### Encode - You found the data in your Clipboard ########################
CompilerIf #PB_Compiler_IsMainFile
  Define len_line=100
  Define password$=""
  Define character_offset=230
  Data_Encoder_AES_BF("", len_line, character_offset, password$)
  MessageRequester("Encoder", "Encode finished"+#LF$+#LF$+ "The data are now in your Clipboard")
CompilerEndIf

Decoder with AES encryption and crypt randomized IV

Code: Select all

DeclareModule Data_Tool_Decoder_AES_BF
  EnableExplicit
  ; Sakis data Generator - © Decoder
  ; Variant with integrated AES encryption and automatically crypt randomized IV
  ; This code is free for use, changing and enhancing
  ; character_offset - Base character offset - As sample 30, 160, 230 - Try what ever you want
  ; Seed - set as a integer - This works as password for the integrated data crypter
  Declare Data_Decoder_AES_BF(*data_start_pointer, *data_end_pointer, password$="")
EndDeclareModule

Module Data_Tool_Decoder_AES_BF
  ; You can remove what you not want
  UsePNGImageDecoder() : UseJPEGImageDecoder() : UseGIFImageDecoder() : UseTIFFImageDecoder() : UseMD5Fingerprint()
  
  Procedure Data_Decoder_AES_BF(*data_start_pointer, *data_end_pointer, password$="")
    ; Sakis data Generator - This function give back a pointer to a allocated memory with your data
    Protected i, ii, i1, depth, breaks, *buffer, *catch_buffer, len_buffer, len_line
    Protected len_data.q, data_size, *catch_buffer_new, string$
    Protected fixed$=StringFingerprint(password$+"%$(s4DäÖÄö", #PB_Cipher_MD5) : Dim register.q(3)
    Repeat ; 16 Bytes
      PokeA(@register(0)+i, Val("$"+PeekS(@fixed$+ii, 2))) : ii+SizeOf(character)<<1 : i+1 ; Create a key
    Until ii=StringByteLength(fixed$) : i=0 : ii=0
    ii=0 : len_line=Len(PeekS(*data_start_pointer))
    data_size=*data_end_pointer-*data_start_pointer : len_buffer=data_size/2
    breaks=data_size/2/len_line : string$=Space(len_buffer+breaks)
    *buffer=AllocateMemory(data_size) : CopyMemory(*data_start_pointer, *buffer, data_size) : i=0 : breaks=0
    Repeat : If Not PeekW(*buffer+i) : PokeW(*buffer+i, 2560) : breaks+1 : EndIf : i+2 : Until i>data_size
    *catch_buffer=AllocateMemory(data_size/2-breaks) : ii=0
    ii=46 : For i=0 To 15 : PokeA(@register(2)+i, PeekA(*data_end_pointer-ii)) : ii-2 : Next : ii=0
    register(0)!register(2) : register(1)!register(3) 
    For i=0 To len_buffer-breaks-1 : If Not i%len_line And i : ii+2 : EndIf
      Repeat
        If Not i1 : AESEncoder(@register(0), @register(0), 16, @register(0), 128, 0, #PB_Cipher_ECB) : EndIf
        depth=PeekA(@register(0)+i1) : i1+1 : If i1>15 : i1=0 : EndIf
      Until depth
      PokeA(*catch_buffer+i, PeekA(*buffer+ii)-depth) : ii+2
    Next
    ii=14 : For i=0 To 5 : PokeA(@len_data+i, PeekA(*data_end_pointer-ii)) : ii-2 : Next
    *catch_buffer_new=AllocateMemory(len_data)
    CopyMemory(*catch_buffer, *catch_buffer_new, MemorySize(*catch_buffer_new))
    FreeMemory(*catch_buffer) : FreeMemory(*buffer)
    ProcedureReturn *catch_buffer_new
  EndProcedure 
EndModule
UseModule Data_Tool_Decoder_AES_BF

; #################### Decoder Demo part ########################
CompilerIf #PB_Compiler_IsMainFile
  EnableExplicit
  
  Define password$=""
  
  Define *catch_buffer=Data_Decoder_AES_BF(?image_1_start, ?image_1_end, password$)
  
  Define image_ID=CatchImage(#PB_Any, *catch_buffer)
  FreeMemory(*catch_buffer)
  
  Define window_ID=OpenWindow(#PB_Any, 0, 0, 600, 400, "", #PB_Window_SystemMenu|#PB_Window_ScreenCentered)
  If IsImage(image_ID)
    Define image_gadget_ID=ImageGadget(#PB_Any,
                                       WindowWidth(window_ID)/2-ImageWidth(image_ID)/2,
                                       WindowHeight(window_ID)/2-ImageHeight(image_ID)/2,
                                       0,
                                       0,
                                       ImageID(image_ID))
  Else
    CloseWindow(window_ID)
    MessageRequester("Hint", "Embedded image not usable")
    End
  EndIf
  
  Repeat : Until WaitWindowEvent()=#PB_Event_CloseWindow
  
  DataSection
    image_1_start:
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
    image_1_end:
  EndDataSection
CompilerEndIf

Encoder - High speed with LZMA compression

Code: Select all

DeclareModule Data_Tool_Encoder_BF
  EnableExplicit
  ; Sakis data Generator © - Encoder - Using StringFields_BF - Incredible fast - With compression
  ; This code is free for use, changing and enhancing
  ; character_offset - Base character offset - As sample 30, 160, 230 - Try what ever you want
  ; Seed - set as a integer - This works as password for the integrated data crypter
  Declare Data_Encoder_BF(path$="", length_data_line=100, character_offset=230, seed=0)
  Declare CompressionPercent_DTE_BF() ; This function show the actual compression as percent
EndDeclareModule

Module Data_Tool_Encoder_BF
  ; StringField_Tool_BF - By Saki - Unicode - This code is free for using and enhancing
  Global NewList index(), NewList indexes.s(), empty_fields.q, start_index.q=1, end_index.q=-1, compression_percent.d
  AddElement(index()) : AddElement(indexes())
  Procedure StringFields_BF(string$, separator$, mode=1, ignore_empty_fields=1)
    Select mode
      Case 0 : Protected add_first=1 : Case 2 : Protected add_last=1 : Case 3 : add_first=1 : add_last=1
    EndSelect
    Protected skip_first : If start_index<1 : start_index=1 : EndIf
    Protected i, ii, iii, iiii, pos_1, pos_2, length_result, comp, count_index, amount_indexes
    Protected len_separator=StringByteLength(separator$), skip_last, *string=@string$
    Protected *separator=@separator$, *pointer.word, byte_pos_last, result$ 
    If end_index=-1 : end_index=1e18 : EndIf : end_index-1
    If start_index>end_index : start_index=end_index+1 : EndIf : If Not PeekW(*string) : ProcedureReturn 0 : EndIf
    ClearList(index()) : AddElement(index()) : ClearList(indexes()) : AddElement(indexes())
    If comp=CompareMemory(*string, *separator, len_separator) : end_index+1 : count_index+1 : skip_first=1 : EndIf
    If add_first
      If skip_first And start_index : start_index-2 : end_index-1 : EndIf
    Else
      If skip_first And start_index : start_index-1 : EndIf 
    EndIf
    i=-2
    Repeat
      i+2 : comp=CompareMemory(*string+i, *separator, len_separator)
      If comp
        iii=i : count_index+1 : ii+1 : i+len_separator-2 : amount_indexes+1 : AddElement(index()) : index()=i+2
      EndIf 
      *pointer=*string+i+len_separator
    Until count_index>end_index Or Not *pointer\w
    iiii=i : byte_pos_last=iii+len_separator
    If comp=CompareMemory(*string+i-len_separator-1, *separator, len_separator) : skip_last=1 : EndIf
    If end_index>count_index : end_index=count_index : EndIf : amount_indexes=ii : i=skip_first
    If start_index>amount_indexes : start_index=amount_indexes-1 : EndIf : i+start_index+skip_first+skip_last
    If skip_first : i-1 : EndIf : If skip_last : i-1 : EndIf
    If amount_indexes
      Repeat 
        If ListSize(index())>i : SelectElement(index(), i) : pos_1=index() : EndIf
        If ListSize(index())>i+1 : SelectElement(index(), i+1) : pos_2=index() : EndIf
        length_result=(pos_2-pos_1-len_separator)>>1
        If pos_2-pos_1>0
          If length_result>0
            result$=PeekS(*string+pos_1, length_result) : AddElement(indexes()) : indexes()=result$ 
          Else 
            empty_fields+1
            If ignore_empty_fields : result$=#Null$ : Else : result$="" : AddElement(indexes()) : EndIf         
          EndIf
        EndIf
        i+1
      Until i>end_index Or i=amount_indexes
      If add_last And skip_last
        result$=PeekS(*string+byte_pos_last, (iiii-byte_pos_last+len_separator)>>1)
        AddElement(indexes()) : indexes()=result$ 
      EndIf
    EndIf
    ProcedureReturn 1
  EndProcedure
  
  Procedure.s GetStringFields_BF(field)
    If field <1 : ProcedureReturn "" : EndIf
    SelectElement(indexes(), field) : If field<ListSize(indexes()) : ProcedureReturn indexes() : EndIf
  EndProcedure
  
  Procedure FreeAllStringFields_BF()
    ClearList(index()) : AddElement(index()) : ClearList(indexes()) : AddElement(indexes())
    empty_fields.q=0 : start_index.q=1 : end_index.q=-1 : ProcedureReturn 1
  EndProcedure
  
  Procedure Data_Encoder_BF(path$="", length_data_line=100, character_offset=230, seed=0)
    ; Sakis data Generator - This code put the generated data in your Clipboard - You see nothing
    Protected i, ii, iii, breaks, *buffer, len_buffer, file, lof_file, len_source.q, padding=3
    Protected string$, new_string$, temp_string$, last$, length$
    Macro Create_length_string
      length$=LSet(Chr(character_offset<<8), 6, Chr(character_offset<<8))
      ii=0 : For i=0 To 5 : PokeA(@length$+ii, PeekA(@len_source+i)) : ii+2 : Next i
    EndMacro
    If path$="" : path$=OpenFileRequester("Select source", "", "", 0) : If path$="" : ProcedureReturn 0 : EndIf : EndIf
    If FileSize(path$)<1 : ProcedureReturn 0 : EndIf
    file=ReadFile(#PB_Any, path$) : If Not file : ProcedureReturn 0 : EndIf
    lof_file=Lof(file) : *buffer=AllocateMemory(lof_file)
    If Not ReadData(file, *buffer, lof_file) : CloseFile(file) : ProcedureReturn 0 : EndIf : CloseFile(file)
    
    RandomSeed(seed)
    UseLZMAPacker()
    Protected length_uncompressed$, length_uncompressed.q=lof_file
    Protected *buffer_compressed=AllocateMemory(lof_file*1.5)
    Protected size_compressed=CompressMemory(*buffer, lof_file, *buffer_compressed,
                                             MemorySize(*buffer_compressed), #PB_PackerPlugin_Lzma)
    If size_compressed<lof_file
      compression_percent=100-100/length_uncompressed*size_compressed
      CopyMemory(*buffer_compressed, *buffer, size_compressed)
      len_buffer=size_compressed+padding
    Else
      compression_percent=-1
      len_buffer=lof_file+padding
    EndIf
    FreeMemory(*buffer_compressed)
    len_source.q=len_buffer-3
    Macro Create_length_uncompressed
      length_uncompressed$=LSet(Chr(character_offset<<8), 6, Chr(character_offset<<8))
      ii=0 : For i=0 To 5 : PokeA(@length_uncompressed$+ii, PeekA(@length_uncompressed+i)) : ii+2 : Next i
    EndMacro
    
    breaks=len_buffer/length_data_line : breaks+Bool(len_buffer%length_data_line) : string$=Space(len_buffer+breaks-1)
    For i=0 To len_buffer-1 : ii+1 : iii+1
      PokeA(@string$+i+ii, character_offset) : PokeA(@string$+i+ii-1, PeekA(*buffer+i)+Random(254, 1))
      If Not iii%length_data_line : ii+1 : PokeW(@string$+i+ii, 10) : ii+1 : EndIf
    Next 
    If breaks<2
      create_length_string : Create_length_uncompressed
      SetClipboardText("Data.s "+Chr(34)+string$+Chr(34)+","+Chr(34)+length_uncompressed$+length$+Chr(34))
      ProcedureReturn 1
    EndIf
    iii=(length_data_line+10) : new_string$=Space(breaks*iii) : ii=0 : iii*2
    StringFields_BF(string$, #LF$, 3)
    For i=1 To breaks
      temp_string$="Data.s "+Chr(34)+GetStringFields_BF(i)+Chr(34)+#LF$
      CopyMemory(@temp_string$, @new_string$+ii, iii) : ii+iii 
    Next
    new_string$+"Data.s "+Chr(34) : Create_length_string : Create_length_uncompressed
    new_string$+length_uncompressed$+length$+Chr(34)
    SetClipboardText(new_string$) : FreeAllStringFields_BF()
    ProcedureReturn 1
  EndProcedure
  
  Procedure CompressionPercent_DTE_BF() : ProcedureReturn compression_percent : EndProcedure
  
EndModule
UseModule Data_Tool_Encoder_BF

; #################### Encode - You found the data in your Clipboard ########################
CompilerIf #PB_Compiler_IsMainFile
  Define len_line=100
  ; Data_Encoder_BF() ; Preset
  Data_Encoder_BF("", len_line)
  MessageRequester("Encoder", "Encode finished"+#LF$+#LF$+ "The data are now in your Clipboard"+#LF$+#LF$+
                              Str(CompressionPercent_DTE_BF())+" % compression")
CompilerEndIf

Decoder - High speed with LZMA compression

Code: Select all

DeclareModule Data_Tool_Decoder_BF
  EnableExplicit
  ; Sakis data Generator - © Decoder - With compression
  ; This code is free for use, changing and enhancing
  ; character_offset - Base character offset - As sample 30, 160, 230 - Try what ever you want
  ; Seed - set as a integer - This works as password for the integrated data crypter
  Declare Data_Decoder_BF(*data_start_pointer, *data_end_pointer, seed=0)
  Declare CompressionPercent_DTD_BF() ; This function show the actual compression as percent
EndDeclareModule

Module Data_Tool_Decoder_BF
  ; You can remove what you not want
  UsePNGImageDecoder() : UseJPEGImageDecoder() : UseGIFImageDecoder() : UseTIFFImageDecoder()
  Global compression_percent.d
  
  Procedure Data_Decoder_BF(*data_start_pointer, *data_end_pointer, seed=0)
    ; Sakis data Generator - This function give back a pointer to a allocated memory with your data
    Protected i, ii, breaks, *buffer, *catch_buffer, len_buffer, len_line
    Protected len_data.q, data_size, *catch_buffer_new
    len_line=Len(PeekS(*data_start_pointer))
    data_size=*data_end_pointer-*data_start_pointer : len_buffer=data_size/2
    RandomSeed(seed) : *buffer=AllocateMemory(data_size) : CopyMemory(*data_start_pointer, *buffer, data_size)
    Repeat : If Not PeekW(*buffer+i) : PokeW(*buffer+i, 2560) : breaks+1 : EndIf : i+2 : Until i>data_size
    *catch_buffer=AllocateMemory(data_size/2-breaks) : ii=0
    For i=0 To len_buffer-breaks-1 : If Not i%len_line And i : ii+2 : EndIf
      PokeA(*catch_buffer+i, PeekA(*buffer+ii)-Random(254, 1)) : ii+2
    Next
    
    Protected len_uncompressed
    ii=14 : For i=0 To 5 : PokeA(@len_data+i, PeekA(*data_end_pointer-ii)) : ii-2 : Next
    ii=26 : For i=0 To 5 : PokeA(@len_uncompressed+i, PeekA(*data_end_pointer-ii)) : ii-2 : Next
    
    *catch_buffer_new=AllocateMemory(len_data)
    CopyMemory(*catch_buffer, *catch_buffer_new, MemorySize(*catch_buffer_new))
    FreeMemory(*catch_buffer) : FreeMemory(*buffer)
    
    UseLZMAPacker()
    Protected *buffer_uncompressed=AllocateMemory(len_uncompressed)
    Protected size_uncompressed=UncompressMemory(*catch_buffer_new, MemorySize(*catch_buffer_new),
                                                 *buffer_uncompressed, MemorySize(*buffer_uncompressed),
                                                 #PB_PackerPlugin_Lzma)
    
    If size_uncompressed<>-1
      Swap *buffer_uncompressed, *catch_buffer_new
      compression_percent=100-100/size_uncompressed*len_data
    Else
      compression_percent=-1
    EndIf
    FreeMemory(*buffer_uncompressed)
    
    ProcedureReturn *catch_buffer_new
  EndProcedure 
  
  Procedure CompressionPercent_DTD_BF() : ProcedureReturn compression_percent : EndProcedure
  
EndModule
UseModule Data_Tool_Decoder_BF

; #################### Decoder Demo part ########################
CompilerIf #PB_Compiler_IsMainFile
  EnableExplicit
  
  Define *catch_buffer=Data_Decoder_BF(?image_1_start, ?image_1_end)
  
  Debug Str(CompressionPercent_DTD_BF())+" % compression" ; This function show the actual compression as percent
  
  Define image_ID=CatchImage(#PB_Any, *catch_buffer)
  FreeMemory(*catch_buffer)
  
  Define window_ID=OpenWindow(#PB_Any, 0, 0, 600, 400, "", #PB_Window_SystemMenu|#PB_Window_ScreenCentered)
  If IsImage(image_ID)
    Define image_gadget_ID=ImageGadget(#PB_Any,
                                       WindowWidth(window_ID)/2-ImageWidth(image_ID)/2,
                                       WindowHeight(window_ID)/2-ImageHeight(image_ID)/2,
                                       0,
                                       0,
                                       ImageID(image_ID))
  Else
    CloseWindow(window_ID)
    MessageRequester("Hint", "Embedded image not usable")
    End
  EndIf
  
  Repeat : Until WaitWindowEvent()=#PB_Event_CloseWindow
  
  DataSection
    image_1_start:
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    Data.s ""
    image_1_end:
  EndDataSection
CompilerEndIf

Encoder with AES encryption, crypt randomized IV and LZMA compression

Code: Select all

DeclareModule Data_Tool_Encoder_AES_BF
  EnableExplicit
  ; Sakis data Generator © - Encoder - Using StringFields_BF - Incredible fast
  ; Variant with integrated AES encryption, automatically crypt randomized IV and compression
  ; This code is free for use, changing and enhancing
  ; character_offset - Base character offset - As sample 30, 160, 230 - Try what ever you want
  ; Seed - set as a integer - This works as password for the integrated data crypter
  Declare Data_Encoder_AES_BF(path$="", length_data_line=100, character_offset=230, password$="")
  Declare CompressionPercent_DTEA_BF() ; this function show the actual compression as percent
EndDeclareModule

Module Data_Tool_Encoder_AES_BF
  UseMD5Fingerprint()
  Global NewList index(), NewList indexes.s(), empty_fields.q, start_index.q=1, end_index.q=-1, compression_percent.d
  AddElement(index()) : AddElement(indexes())
  Procedure StringFields_BF(string$, separator$, mode=1, ignore_empty_fields=1)
    Select mode
      Case 0 : Protected add_first=1 : Case 2 : Protected add_last=1 : Case 3 : add_first=1 : add_last=1
    EndSelect
    Protected skip_first : If start_index<1 : start_index=1 : EndIf
    Protected i, ii, iii, iiii, pos_1, pos_2, length_result, comp, count_index, amount_indexes
    Protected len_separator=StringByteLength(separator$), skip_last, *string=@string$
    Protected *separator=@separator$, *pointer.word, byte_pos_last, result$ 
    If end_index=-1 : end_index=1e18 : EndIf : end_index-1
    If start_index>end_index : start_index=end_index+1 : EndIf : If Not PeekW(*string) : ProcedureReturn 0 : EndIf
    ClearList(index()) : AddElement(index()) : ClearList(indexes()) : AddElement(indexes())
    If comp=CompareMemory(*string, *separator, len_separator) : end_index+1 : count_index+1 : skip_first=1 : EndIf
    If add_first
      If skip_first And start_index : start_index-2 : end_index-1 : EndIf
    Else
      If skip_first And start_index : start_index-1 : EndIf 
    EndIf
    i=-2
    Repeat
      i+2 : comp=CompareMemory(*string+i, *separator, len_separator)
      If comp
        iii=i : count_index+1 : ii+1 : i+len_separator-2 : amount_indexes+1 : AddElement(index()) : index()=i+2
      EndIf 
      *pointer=*string+i+len_separator
    Until count_index>end_index Or Not *pointer\w
    iiii=i : byte_pos_last=iii+len_separator
    If comp=CompareMemory(*string+i-len_separator-1, *separator, len_separator) : skip_last=1 : EndIf
    If end_index>count_index : end_index=count_index : EndIf : amount_indexes=ii : i=skip_first
    If start_index>amount_indexes : start_index=amount_indexes-1 : EndIf : i+start_index+skip_first+skip_last
    If skip_first : i-1 : EndIf : If skip_last : i-1 : EndIf
    If amount_indexes
      Repeat 
        If ListSize(index())>i : SelectElement(index(), i) : pos_1=index() : EndIf
        If ListSize(index())>i+1 : SelectElement(index(), i+1) : pos_2=index() : EndIf
        length_result=(pos_2-pos_1-len_separator)>>1
        If pos_2-pos_1>0
          If length_result>0
            result$=PeekS(*string+pos_1, length_result) : AddElement(indexes()) : indexes()=result$ 
          Else 
            empty_fields+1
            If ignore_empty_fields : result$=#Null$ : Else : result$="" : AddElement(indexes()) : EndIf         
          EndIf
        EndIf
        i+1
      Until i>end_index Or i=amount_indexes
      If add_last And skip_last
        result$=PeekS(*string+byte_pos_last, (iiii-byte_pos_last+len_separator)>>1)
        AddElement(indexes()) : indexes()=result$ 
      EndIf
    EndIf
    ProcedureReturn 1
  EndProcedure
  
  Procedure.s GetStringFields_BF(field)
    If field <1 : ProcedureReturn "" : EndIf
    SelectElement(indexes(), field) : If field<ListSize(indexes()) : ProcedureReturn indexes() : EndIf
  EndProcedure
  
  Procedure FreeAllStringFields_BF()
    ClearList(index()) : AddElement(index()) : ClearList(indexes()) : AddElement(indexes())
    empty_fields.q=0 : start_index.q=1 : end_index.q=-1 : ProcedureReturn 1
  EndProcedure
  
  Procedure Data_Encoder_AES_BF(path$="", length_data_line=100, character_offset=230, password$="")
    ; Sakis data Generator - This code put the generated data in your Clipboard - You see nothing
    Protected i, ii, iii, i1, depth, breaks, *buffer, len_buffer, file, lof_file, len_source.q, padding=3
    Protected string$, new_string$, temp_string$, last$, length$, iv$
    Protected fixed$=StringFingerprint(password$+"%$(s4DäÖÄö", #PB_Cipher_MD5) : Dim register.q(3)
    Repeat ; 16 Bytes
      PokeA(@register(0)+i, Val("$"+PeekS(@fixed$+ii, 2))) : ii+SizeOf(character)<<1 : i+1 ; Create a key
    Until ii=StringByteLength(fixed$) : i=0 : ii=0
    If OpenCryptRandom() : CryptRandomData(@register(2), 16) : Else : RandomData(@register(2), 16) : EndIf
    register(0)!register(2) : register(1)!register(3) 
    Macro Create_length_string
      length$=LSet(Chr(character_offset<<8), 6, Chr(character_offset<<8))
      ii=0 : For i=0 To 5 : PokeA(@length$+ii, PeekA(@len_source+i)) : ii+2 : Next i
    EndMacro
    Macro Create_string
      iv$=LSet(Chr(character_offset<<8), 16, Chr(character_offset<<8))
      ii=0 : For i=0 To 15 : PokeA(@iv$+ii, PeekA(@register(2)+i)) : ii+2 : Next i
    EndMacro
    If path$="" : path$=OpenFileRequester("Select source", "", "", 0) : If path$="" : ProcedureReturn 0 : EndIf : EndIf
    If FileSize(path$)<1 : ProcedureReturn 0 : EndIf
    file=ReadFile(#PB_Any, path$) : If Not file : ProcedureReturn 0 : EndIf
    lof_file=Lof(file) : *buffer=AllocateMemory(lof_file)
    If Not ReadData(file, *buffer, lof_file) : CloseFile(file) : ProcedureReturn 0 : EndIf : CloseFile(file)
    
    UseLZMAPacker()
    Protected length_uncompressed$, length_uncompressed.q=lof_file
    Protected *buffer_compressed=AllocateMemory(lof_file*1.5)
    Protected size_compressed=CompressMemory(*buffer, lof_file, *buffer_compressed,
                                             MemorySize(*buffer_compressed), #PB_PackerPlugin_Lzma)
    If size_compressed<lof_file
      compression_percent=100-100/length_uncompressed*size_compressed
      CopyMemory(*buffer_compressed, *buffer, size_compressed)
      len_buffer=size_compressed+padding
    Else
      compression_percent=-1
      len_buffer=lof_file+padding
    EndIf
    FreeMemory(*buffer_compressed)
    len_source.q=len_buffer-3
    Macro Create_length_uncompressed
      length_uncompressed$=LSet(Chr(character_offset<<8), 6, Chr(character_offset<<8))
      ii=0 : For i=0 To 5 : PokeA(@length_uncompressed$+ii, PeekA(@length_uncompressed+i)) : ii+2 : Next i
    EndMacro
    
    breaks=len_buffer/length_data_line : breaks+Bool(len_buffer%length_data_line) : string$=Space(len_buffer+breaks-1)
    For i=0 To len_buffer-1 : ii+1 : iii+1
      Repeat
        If Not i1 : AESEncoder(@register(0), @register(0), 16, @register(0), 128, 0, #PB_Cipher_ECB) : EndIf
        depth=PeekA(@register(0)+i1) : i1+1 : If i1>15 : i1=0 : EndIf
      Until depth
      PokeA(@string$+i+ii, character_offset) : PokeA(@string$+i+ii-1, PeekA(*buffer+i)+depth)
      If Not iii%length_data_line : ii+1 : PokeW(@string$+i+ii, 10) : ii+1 : EndIf
    Next 
    If breaks<2
      Create_length_string : Create_string : Create_length_uncompressed
      SetClipboardText("Data.s "+Chr(34)+string$+Chr(34)+","+Chr(34)+iv$+length_uncompressed$+length$+Chr(34))
      ProcedureReturn 1
    EndIf
    iii=(length_data_line+10) : new_string$=Space(breaks*iii) : ii=0 : iii*2
    StringFields_BF(string$, #LF$, 3)
    For i=1 To breaks
      temp_string$="Data.s "+Chr(34)+GetStringFields_BF(i)+Chr(34)+#LF$
      CopyMemory(@temp_string$, @new_string$+ii, iii) : ii+iii 
    Next
    new_string$+"Data.s "+Chr(34) : Create_length_string : Create_string : Create_length_uncompressed
    new_string$+iv$+length_uncompressed$+length$+Chr(34)
    SetClipboardText(new_string$) : FreeAllStringFields_BF()
    ProcedureReturn 1
  EndProcedure
  
  Procedure CompressionPercent_DTEA_BF() : ProcedureReturn compression_percent : EndProcedure
  
EndModule
UseModule Data_Tool_Encoder_AES_BF

; #################### Encode - You found the data in your Clipboard ########################
CompilerIf #PB_Compiler_IsMainFile
  Define len_line=100
  Define password$=""
  Define character_offset=230
  Data_Encoder_AES_BF("", len_line, character_offset, password$)
  MessageRequester("Encoder", "Encode finished"+#LF$+#LF$+ "The data are now in your Clipboard"+#LF$+#LF$+
                              Str(CompressionPercent_DTEA_BF())+" % compression")
CompilerEndIf

Decoder with AES encryption, crypt randomized IV and LZMA compression

Code: Select all

DeclareModule Data_Tool_Decoder_AES_BF
  EnableExplicit
  ; Sakis data Generator - © Decoder
  ; Variant with integrated AES encryption, automatically crypt randomized IV and compression
  ; This code is free for use, changing and enhancing
  ; character_offset - Base character offset - As sample 30, 160, 230 - Try what ever you want
  ; Seed - set as a integer - This works as password for the integrated data crypter
  Declare Data_Decoder_AES_BF(*data_start_pointer, *data_end_pointer, password$="")
  Declare CompressionPercent_DTDA_BF() ; This function show the actual compression as percent
EndDeclareModule

Module Data_Tool_Decoder_AES_BF
  ; You can remove what you not want
  UsePNGImageDecoder() : UseJPEGImageDecoder() : UseGIFImageDecoder() : UseTIFFImageDecoder() : UseMD5Fingerprint()
  Global compression_percent.d
  
  Procedure Data_Decoder_AES_BF(*data_start_pointer, *data_end_pointer, password$="")
    ; Sakis data Generator - This function give back a pointer to a allocated memory with your data
    Protected i, ii, i1, depth, breaks, *buffer, *catch_buffer, len_buffer, len_line
    Protected len_data.q, data_size, *catch_buffer_new
    Protected fixed$=StringFingerprint(password$+"%$(s4DäÖÄö", #PB_Cipher_MD5) : Dim register.q(3)
    Repeat ; 16 Bytes
      PokeA(@register(0)+i, Val("$"+PeekS(@fixed$+ii, 2))) : ii+SizeOf(character)<<1 : i+1 ; Create a key
    Until ii=StringByteLength(fixed$) : i=0 : ii=0
    len_line=Len(PeekS(*data_start_pointer))
    data_size=*data_end_pointer-*data_start_pointer : len_buffer=data_size/2
    *buffer=AllocateMemory(data_size) : CopyMemory(*data_start_pointer, *buffer, data_size) : i=0
    Repeat : If Not PeekW(*buffer+i) : PokeW(*buffer+i, 2560) : breaks+1 : EndIf : i+2 : Until i>data_size
    *catch_buffer=AllocateMemory(data_size/2-breaks)
    ii=58 : For i=0 To 15 : PokeA(@register(2)+i, PeekA(*data_end_pointer-ii)) : ii-2 : Next : ii=0
    register(0)!register(2) : register(1)!register(3) 
    For i=0 To len_buffer-breaks-1 : If Not i%len_line And i : ii+2 : EndIf
      Repeat
        If Not i1 : AESEncoder(@register(0), @register(0), 16, @register(0), 128, 0, #PB_Cipher_ECB) : EndIf
        depth=PeekA(@register(0)+i1) : i1+1 : If i1>15 : i1=0 : EndIf
      Until depth
      PokeA(*catch_buffer+i, PeekA(*buffer+ii)-depth) : ii+2
    Next
    
    Protected len_uncompressed
    ii=14 : For i=0 To 5 : PokeA(@len_data+i, PeekA(*data_end_pointer-ii)) : ii-2 : Next
    ii=26 : For i=0 To 5 : PokeA(@len_uncompressed+i, PeekA(*data_end_pointer-ii)) : ii-2 : Next
    
    *catch_buffer_new=AllocateMemory(len_data)
    CopyMemory(*catch_buffer, *catch_buffer_new, MemorySize(*catch_buffer_new))
    FreeMemory(*catch_buffer) : FreeMemory(*buffer)
    
    UseLZMAPacker()
    Protected *buffer_uncompressed=AllocateMemory(len_uncompressed)
    Protected size_uncompressed=UncompressMemory(*catch_buffer_new, MemorySize(*catch_buffer_new),
                                                 *buffer_uncompressed, MemorySize(*buffer_uncompressed),
                                                 #PB_PackerPlugin_Lzma)
    
    If size_uncompressed<>-1
      Swap *buffer_uncompressed, *catch_buffer_new
      compression_percent=100-100/size_uncompressed*len_data
    Else
      compression_percent=-1
    EndIf
    FreeMemory(*buffer_uncompressed)
    
    ProcedureReturn *catch_buffer_new
  EndProcedure 
  
  Procedure CompressionPercent_DTDA_BF() : ProcedureReturn compression_percent : EndProcedure
  
EndModule
UseModule Data_Tool_Decoder_AES_BF

; #################### Decoder Demo part ########################
CompilerIf #PB_Compiler_IsMainFile
  EnableExplicit
  
  Define password$=""
  
  Define *catch_buffer=Data_Decoder_AES_BF(?image_1_start, ?image_1_end, password$)
  
  Debug Str(CompressionPercent_DTDA_BF())+" % compression" ; This function show the actual compression as percent
  
  Define image_ID=CatchImage(#PB_Any, *catch_buffer)
  FreeMemory(*catch_buffer)
  
  Define window_ID=OpenWindow(#PB_Any, 0, 0, 600, 400, "", #PB_Window_SystemMenu|#PB_Window_ScreenCentered)
  If IsImage(image_ID)
    Define image_gadget_ID=ImageGadget(#PB_Any,
                                       WindowWidth(window_ID)/2-ImageWidth(image_ID)/2,
                                       WindowHeight(window_ID)/2-ImageHeight(image_ID)/2,
                                       0,
                                       0,
                                       ImageID(image_ID))
  Else
    CloseWindow(window_ID)
    MessageRequester("Hint", "Embedded image not usable")
    End
  EndIf
  
  Repeat : Until WaitWindowEvent()=#PB_Event_CloseWindow
  
  DataSection
    image_1_start:
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
Data.s ""
    image_1_end:
  EndDataSection
CompilerEndIf
Last edited by Saki on Fri Jun 25, 2021 7:28 pm, edited 74 times in total.
地球上の平和
User avatar
idle
Always Here
Always Here
Posts: 5042
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by idle »

So the idea is to use a single 16 bit Unicode character to store a byte in it.
ummm, you're saying that you want to store 8 bits in 16 bits? I think you mean the opposite of that that.

Whats the point of this in a datasection. Do you want a shorter section? Then Just use hex and quads and if you want the actual data smaller as well use lzma or deflate if the data is compressible.
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by Saki »

This is not easy to understand.

PNG and JPEG are usually no longer additionally compressible or even become larger as a result.

Compare :
Above standard 40 Bytes in one row.
Below mine, 100 Bytes in one row.
Image

Read something here (below)

https://www.purebasic.fr/english/viewto ... 14&t=77246
Last edited by Saki on Mon May 17, 2021 9:05 pm, edited 1 time in total.
地球上の平和
User avatar
idle
Always Here
Always Here
Posts: 5042
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by idle »

Ok I see what you're doing, clever trick but not really practical as it results in a binary datasection twice the size.
User avatar
Keya
Addict
Addict
Posts: 1891
Joined: Thu Jun 04, 2015 7:10 am

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by Keya »

just to clarify, is this about making the source code smaller, or the compiled data?
User avatar
idle
Always Here
Always Here
Posts: 5042
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by idle »

Keya wrote: Mon May 17, 2021 1:29 am just to clarify, is this about making the source code smaller, or the compiled data?
It's about making the source code smaller
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by Saki »

Hi,
Yes, it is.
Practically, however, it is not relevant if the binary data section is enlarged by this.
The amount of binary data in these data sections is very small.
Only the source codes are enormously inflated by data sections.
With my code, the size of the data section in the source code can be reduced to 50%.
This makes the data sections in the source code look very compact and clean and they are much easier to keep track of.

It is easily overlooked today that the use of Unicode has the effect of
the size of the binary data is doubled without any practical benefit.

If Base64 is used, the overhead is even about 35% greater than with my solution.
But nobody is bothered by this, it is accepted because it has no noticeable effect.
Most of the time people are simply not aware of the existence of this fact.

Datas in these data areas is a popular point of attack.
They can be easily manipulated.
Images, logos, copyright notices or even code can be exchanged or removed.

My solution diffuses these critical areas in such a way that an exchange of data is
becomes very difficult for any stranger, because embedded images and data can no longer be located
if you do not know exactly what you are looking at.
Even if you understand it, it's not much fun trying to deal with it.

Practically you can see that here in the forum it was already very difficult to make clear what is actually happening.

The images or icons embedded in the data lines are usually tiny or small, and rarely
larger.
However, they inflate the data area in the source code enormously.

If we assume a size of 20kb in JPEG or PNG format for the images to be embedded,
we get 512 data lines in the source code, as sample, with the standard solution from ts-soft.
With my solution with an average width of 100 bytes per line, it is only 205 lines.
In binary terms, this results in 20kb of data plus 20kb of diffusion with my solution.
These 20kb diffusion are not rubbish in themselves, but secure the embedding from manipulation.

Whether an executable becomes 20kb larger by this measure or not is of no importance at all.
In almost every executable there is usually a lot of unnecessary rubbish that is many times larger.
In principle, it is no different from entering a unicode string in a data line.

The difference between rubbish and ballast is considerable, rubbish is always rubbish, ballast is useful and often indispensable.
Last edited by Saki on Mon May 17, 2021 2:17 pm, edited 2 times in total.
地球上の平和
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by Saki »

Code updated and enhanced.

Yesterday evening I compressed the code a bit.
In the process, I introduced a bug in the encoder due to a typing error.

Now the encoder works again.
Simplifications for using.
Speed up encoder.
A small tutorial has been added in the first post.

The new code now encrypts the data automatically.
You don't need a password, but you can set a seed as password.
The seed is an integer (Remember that integers for x86 can only be four bytes in size).
When the seed is set, the encryption changes absolutely.
This means that, for example, pictures are also ever automatically encrypted.
You yourself see absolutely nothing of this, it happens completely in the background.

This now offers a very high level of security against manipulation,
because the whole process is very difficult to see through.
In addition, everything appears extremely incomprehensible to an attacker,
because he can't find anything usable in the data part.
地球上の平和
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by Saki »

New High Speed Encoder added

This new encoder uses instead of the PB StringField() function
my own StringFields_BF() function.
https://www.purebasic.fr/english/viewto ... 12&t=77219

This increases the speed by more than a hundredfold.

It can produce about 100,000 lines in one second on my machine instead of about 800 lines in one second.

Have fun with it
地球上の平和
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by Saki »

Codes updated and enhanced

The calculation of the last required byte and an automatic padding has been revised.

The latest version of the StringField_BF inserted.
地球上の平和
User avatar
STARGÅTE
Addict
Addict
Posts: 2067
Joined: Thu Jan 10, 2008 1:30 pm
Location: Germany, Glienicke
Contact:

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by STARGÅTE »

I still can't follow your statement "50% smaller output".

Usually we can store binary data in a data section with hexadecimal numbers and use 2 characters (2 bytes in UTF-8) in the source code for each single binary byte.
In your module you are using characters in the unicode range which needs a single character but 3 bytes (UTF-8) in in the source code for each single binary byte.
Of cause, it looks smaller, because these 3 bytes are displayed as a single character, but it is not smaller. You need more memory space in the source code as well as more memory space in the forum.

So from my point of view, the source code is 50% larger, and I can't see the advantage.
PB 6.01 ― Win 10, 21H2 ― Ryzen 9 3900X, 32 GB ― NVIDIA GeForce RTX 3080 ― Vivaldi 6.0 ― www.unionbytes.de
Lizard - Script language for symbolic calculations and moreTypeface - Sprite-based font include/module
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: DATA_Tool_BF - Data Generator - 50% smaller output - Module

Post by Saki »

Well, it works like a memory cell.
The unicode characters are 2 bytes in size and can hold one byte each.
Hex dumps need 2 characters to represent one byte, but visually need more than twice the space in the source code.
Base64 needs about 1.35 times the space.
The advantage is that your source code appears much more compact because the data sections are only half the size.
The term 50% narrower output may be confusing, but I don't know how to write it any other way,
since it factually always needs an explanation.

Compare the codes in your IDE,
how it is in the forum is not so important, there you see now anyway only a few lines of code.
And the limitations in the forum we know, it's just so, not much goes in, but I think still enough.
It is also not the way to post huge codes, i self think.

Another advantage is that your embedded data is always automatically encrypted in your exe.
A simple but here very efficient method to protect the data from manipulation and exploration.
地球上の平和
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Post by Saki »

Temporary decoder bug removed on all decoders.

New codes added.

A variant with supporting AES and a fully automatic randomized IV has been added.
This means that all encrypted data is encrypted completely differently for each encryption process.
The behavior of the crypter is similar to the CBC mode with randomized IV.
地球上の平和
User avatar
Saki
Addict
Addict
Posts: 830
Joined: Sun Apr 05, 2020 11:28 am
Location: Pandora

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Post by Saki »

Since this is asked more often.

if you want to insert encrypted text into the Date section you need to do almost nothing at all.

It is incredibly simple. :wink:

Create the text with a simple text editor.
Save the created MyText.txt file.

Encode this file with the Data Encoder.
Paste the generated code from your clipboard into the Data section of your code.
That's all !

Read it out like this :

Code: Select all

  Define *catch_buffer=Data_Decoder_BF(?text_1_start, ?text_1_end)
  Define text$=PeekS(*catch_buffer, MemorySize(*catch_buffer), #PB_UTF8)
  Debug text$
The data coders always encrypt automatically.
This process itself is unusual and the texts are almost impossible to locate, so to get a solid protection you don't even need to set a password or a seed.
It is also not enough for an attacker to simply replace the text, because the decoder will then encrypt it instead of decrypting it.

The idea of the encryption methods used here is that it can make the encryption almost ineffective if a password has to be stored in the code.
The seed for example is difficult to recognize as a password, it is simply an integer variable.

The methodology shown here can also be used with the InCode Data Tool
地球上の平和
User avatar
Keya
Addict
Addict
Posts: 1891
Joined: Thu Jun 04, 2015 7:10 am

Re: Data_Tool_BF - Data Generator - 50% smaller output + AES - Modules

Post by Keya »

By using PeekS you're creating a second copy of the buffer. You can just use the one buffer:

Code: Select all

 Text$ = "test"
 *catch_buffer = @Text$
 *catch_bufstr.String = @*catch_buffer
 Debug *catch_bufstr\s   
Post Reply