Page 1 of 3

PureLZMA : compress/uncompress data using LZMA algo

Posted: Mon Jan 05, 2004 4:51 pm
by gnozal
PureLZMA library
PureLZMA packer library

Overview

Code: Select all

PureLZMA provides two functions to compress/uncompress data using the LZMA algorithm.
Optional RC4 based encryption is supported.
PureLZMA-Packer provides additional archiver functions.
It uses unmodified code from the LZMA SDK v4.62.
Functions
Basic -> PB4.xx : http://gnozal.ucoz.com/PureLZMA_.htm
Archiver -> PB4.3x+ : http://gnozal.ucoz.com/PureLZMA-Packer.htm
Archiver - Mem -> PB4.3x+ : http://gnozal.ucoz.com/PureLZMA-Packer-Mem.htm

Example 1

Code: Select all

TestFile$ = #PB_Compiler_Home + "PureBasic.exe"
;
Debug "----------- COMPRESS"
;
;
; Load original file into buffer
Stream = ReadFile(#PB_Any, TestFile$)
If Stream
  BufferLen = Lof(Stream)
  *Buffer =  AllocateMemory(BufferLen)
  If *Buffer = 0
    End
  EndIf
  ReadData(Stream, *Buffer, BufferLen)
  CloseFile(Stream)
Else
  Debug "Could not read original file"
  End
EndIf
;
;
; Compress buffer with LZMA
*OutBuffer = PureLZMA_Compress(*Buffer, @BufferLen)
Debug "PureLZMA_Compress : Returned value = " + Str(*OutBuffer)
Debug "PureLZMA_Compress : BufferLen = " + Str(BufferLen)
;
; PureLZMA_Compress() error codes :
;   -1  :  Memory allocation error
;   -2  :  Output buffer overflow or buffer not compressable
;   -3  :  Other compression error
; 
FreeMemory(*Buffer)
;
;
; Save compressed buffer in '.lzma' file
If *OutBuffer > 0
  Stream = CreateFile(#PB_Any, TestFile$ + ".lzma")
  If Stream
    WriteData(Stream, *OutBuffer, BufferLen)
    CloseFile(Stream)
  EndIf
  PureLZMA_FreeBuffer(*OutBuffer)
Else
  Debug "PureLZMA_Compress() failed, error = " + Str(*OutBuffer)
  End
EndIf
;
;
;
Debug "----------- UNCOMPRESS"
;
;
; Load compressed buffer from '.lzma' file
Stream = ReadFile(#PB_Any, TestFile$ + ".lzma")
If Stream
  BufferLen = Lof(Stream)
  *Buffer = AllocateMemory(BufferLen)
  If *Buffer = 0
    End
  EndIf
  ReadData(Stream, *Buffer, BufferLen)
  CloseFile(Stream)
Else
  Debug "Could not read compressed file"
  End
EndIf
;
;
; Unpack buffer with LZMA
*OutBuffer = PureLZMA_UnCompress(*Buffer, @BufferLen)
Debug "PureLZMA_UnCompress : Returned value = " + Str(*OutBuffer)
Debug "PureLZMA_UnCompress : BufferLen = " + Str(BufferLen)
;
; PureLZMA_UnCompress() error codes :
;    0    : Invalid data [not compressed by PureLZMA_Compress() ?]
;   -1    : Memory allocation error
;   -2    : Input buffer insufficient
;   -3,-4 : Other compression error
;
FreeMemory(*Buffer)
;
;
; Save unpacked buffer in '.lzma.unpacked' file
If *OutBuffer > 0
  Stream = CreateFile(#PB_Any, TestFile$ + ".lzma.unpacked")
  If Stream
    WriteData(Stream,*OutBuffer, BufferLen)
    CloseFile(Stream)
  EndIf
  PureLZMA_FreeBuffer(*OutBuffer)
Else
  Debug "PureLZMA_UnCompress() failed, error = " + Str(*OutBuffer)
  End
EndIf
;
; Compare original and '.lzma.unpacked' files !
;
Debug "----------- RESULT"
If MD5FileFingerprint(TestFile$) = MD5FileFingerprint(TestFile$ + ".lzma.unpacked")
  Debug "Compression with PureLZMA is Ok !"
Else
  Debug "PureLZMA failed !"
EndIf
Example 2

Code: Select all

If PureLZMA_Archive_Create(#PB_Compiler_Home + "\Examples\PureLZMA\TestArchive.arc")
  PureLZMA_Archive_Compress(#PB_Compiler_Home + "Examples\Sources\Screen3DRequester.pb", 0)
  PureLZMA_Archive_Compress(#PB_Compiler_Home + "Examples\Sources\MDI ImageViewer.pb", 0)
  PureLZMA_Archive_Compress(#PB_Compiler_Home + "Examples\Sources\Data\Background.bmp", 0)
  PureLZMA_Archive_Close()
EndIf
;
If PureLZMA_Archive_Read(#PB_Compiler_Home + "\Examples\PureLZMA\TestArchive.arc")
  ArchiveInfo.LZMA_ArchiveInfo
  Status = PureLZMA_Archive_FindFirst()
  ;
  While Status
    ;
    If PureLZMA_Archive_GetArchiveInfo(@ArchiveInfo)
      Debug "----------------"
      Debug "CRCUnpacked    = $" + RSet(Hex(ArchiveInfo\CRCUnpacked, #PB_Long), 8, "0")
      Debug "CRCPacked      = $" + RSet(Hex(ArchiveInfo\CRCPacked, #PB_Long), 8, "0")
      Debug "CRCPacked CALC = $" + RSet(Hex(PureLZMA_Archive_GetCRC(), #PB_Long), 8, "0")
      Debug "LengthUnpacked = " + Str(ArchiveInfo\LengthUnpacked)
      Debug "LengthPacked   = " + Str(ArchiveInfo\LengthPacked) + " (" + StrF(ArchiveInfo\LengthPacked / ArchiveInfo\LengthUnpacked * 100, 2) + "% of original)"
      Debug "LZMAFlags      = " + Str(ArchiveInfo\LZMAFlags)
      Debug "FileAttributes = $" + Hex(ArchiveInfo\FileAttributes)
      Debug "Filename       = '" + ArchiveInfo\Filename + "'"
    EndIf
    ;
    Status = PureLZMA_Archive_FindNext()
    ;
  Wend
  ;
  PureLZMA_Archive_Close()
EndIf
Download
Only available for Purebasic Windows x86
PB4.0x-4.2x : http://gnozal.ucoz.com/PureLZMA_.zip
PB4.3x : http://gnozal.ucoz.com/PureLZMA_430.zip
PB4.4x : http://gnozal.ucoz.com/PureLZMA_440.zip
PB4.5x : http://gnozal.ucoz.com/PureLZMA_450.zip
PB4.6x : http://gnozal.ucoz.com/PureLZMA_460.zip
PB5.0x : http://gnozal.ucoz.com/PureLZMA_500.zip
Shared library (DLL) : http://gnozal.ucoz.com/PureLZMA_DLL.zip

Note : although the library uses the LZMA algorithm, which is one of many supported by 7-ZIP, it does no provide support for 7-ZIP archives.

Posted: Thu Dec 18, 2008 9:54 am
by graves
Hi, gnozal,
Explendid library!

From your PureLZMA help:
The functions use the default settings (level / dictSize / lc / lp / pb / fb) for compression.
Would it be possible to extend it to include the compression level at least?
(as 7Zip Plugin for TotalCommander)

Posted: Thu Dec 18, 2008 2:29 pm
by AND51
Great lib, thanks for sharing! :D

Posted: Mon Dec 22, 2008 2:20 pm
by gnozal
graves wrote:
The functions use the default settings (level / dictSize / lc / lp / pb / fb) for compression.
Would it be possible to extend it to include the compression level at least? (as 7Zip Plugin for TotalCommander)
I guess I could add some PureLZMA_SetOptions() function for the next version.

Posted: Mon Dec 22, 2008 4:45 pm
by graves
Thanks, gnozal.
I'm using it in my SaveRestore program for mysql DataBase, and archives with PureLZMA reduces up to 50% over archives with PB native compression. (i.e. from 132 MB to 65 MB for a DataBase of 1,2 GB approx.)

Posted: Tue Dec 23, 2008 9:04 am
by gnozal
Update

Added new function : PureLZMA_SetOptions(Opt_Level, DictSize, lc, lp, pb, fb, NumThreads)

Posted: Tue Dec 23, 2008 9:59 am
by graves
OK, McKay!

Reduced from 65 MB to 60 MB using compression level 7.

Thanks again.

Posted: Sat Jan 10, 2009 11:22 am
by gnozal
Update

Changes :
- added PureLZMA-Packer library, which adds archiver functions to PureLZMA (uses PureLZMA for compression).

Code example :

Code: Select all

If PureLZMA_Archive_Create(#PB_Compiler_Home + "\Examples\PureLZMA\TestArchive.arc")
  PureLZMA_Archive_Compress(#PB_Compiler_Home + "Examples\Sources\Screen3DRequester.pb", 0)
  PureLZMA_Archive_Compress(#PB_Compiler_Home + "Examples\Sources\MDI ImageViewer.pb", 0)
  PureLZMA_Archive_Compress(#PB_Compiler_Home + "Examples\Sources\Data\Background.bmp", 0)
  PureLZMA_Archive_Close()
EndIf
;
If PureLZMA_Archive_Read(#PB_Compiler_Home + "\Examples\PureLZMA\TestArchive.arc")
  ArchiveInfo.LZMA_ArchiveInfo
  If PureLZMA_Archive_FindFirst()
    ;
    Debug PureLZMA_Archive_GetArchiveInfo(@ArchiveInfo)
    Debug "----------------"
    Debug "CRCUnpacked    = $" + RSet(Hex(ArchiveInfo\CRCUnpacked, #PB_Long), 8, "0")
    Debug "CRCPacked      = $" + RSet(Hex(ArchiveInfo\CRCPacked, #PB_Long), 8, "0")
    Debug "CRCPacked CALC = $" + RSet(Hex(PureLZMA_Archive_GetCRC(), #PB_Long), 8, "0")
    Debug "LengthUnpacked = " + Str(ArchiveInfo\LengthUnpacked)
    Debug "LengthPacked   = " + Str(ArchiveInfo\LengthPacked) + " (" + StrF(ArchiveInfo\LengthPacked / ArchiveInfo\LengthUnpacked * 100, 2) + "% of original)"
    Debug "LZMAFlags      = " + Str(ArchiveInfo\LZMAFlags)
    Debug "FileAttributes = $" + Hex(ArchiveInfo\FileAttributes)
    Debug "Filename       = '" + ArchiveInfo\Filename + "'"
    ;
    PureLZMA_Archive_Extract(#PB_Compiler_Home + "\Examples\PureLZMA\" + ArchiveInfo\Filename)
    ;
    While PureLZMA_Archive_FindNext()
      ;
      Debug PureLZMA_Archive_GetArchiveInfo(@ArchiveInfo)
      Debug "----------------"
      Debug "CRCUnpacked    = $" + RSet(Hex(ArchiveInfo\CRCUnpacked, #PB_Long), 8, "0")
      Debug "CRCPacked      = $" + RSet(Hex(ArchiveInfo\CRCPacked, #PB_Long), 8, "0")
      Debug "CRCPacked CALC = $" + RSet(Hex(PureLZMA_Archive_GetCRC(), #PB_Long), 8, "0")
      Debug "LengthUnpacked = " + Str(ArchiveInfo\LengthUnpacked)
      Debug "LengthPacked   = " + Str(ArchiveInfo\LengthPacked) + " (" + StrF(ArchiveInfo\LengthPacked / ArchiveInfo\LengthUnpacked * 100, 2) + "% of original)"
      Debug "LZMAFlags      = " + Str(ArchiveInfo\LZMAFlags)
      Debug "FileAttributes = $" + Hex(ArchiveInfo\FileAttributes)
      Debug "Filename       = '" + ArchiveInfo\Filename + "'"
      ;
      PureLZMA_Archive_Extract(#PB_Compiler_Home + "\Examples\PureLZMA\" + ArchiveInfo\Filename)
      ;
    Wend
  EndIf
  ;
  PureLZMA_Archive_Close()
EndIf

Posted: Tue Jan 13, 2009 1:31 pm
by graves
Hi, gnozal

That's a pretty enhancement.
I'm tested it, and I'm found a problem:
If I create an archive with two or more files into, I'm not could open it again to display the file list.

"if PureLZMA_Archive_Read(filename)" always return #Null.

Also, I need to create a PureLZMA_Archive_CompressMem before my files. I cannot open the archive again.

Posted: Tue Jan 13, 2009 1:41 pm
by gnozal
Hi graves, does the code example below work for you ?

Code: Select all

If PureLZMA_Archive_Create(#PB_Compiler_Home + "\Examples\PureLZMA\TestArchive.arc") 
  TestBuffer.s = "Line 1 ; 11111111111111111111111" + Chr(10) + "Line 2 ; 2222222222222222222222222222"
  PureLZMA_Archive_CompressMem(@TestBuffer, Len(TestBuffer), "PureLZMA_Archive_CompressMemTest")
  PureLZMA_Archive_Compress(#PB_Compiler_Home + "Examples\Sources\Screen3DRequester.pb", 0) 
  PureLZMA_Archive_Compress(#PB_Compiler_Home + "Examples\Sources\MDI ImageViewer.pb", 0) 
  PureLZMA_Archive_Compress(#PB_Compiler_Home + "Examples\Sources\Data\Background.bmp", 0) 
  PureLZMA_Archive_Close() 
EndIf 
; 
If PureLZMA_Archive_Read(#PB_Compiler_Home + "\Examples\PureLZMA\TestArchive.arc") 
  ArchiveInfo.LZMA_ArchiveInfo 
  If PureLZMA_Archive_FindFirst() 
    ; 
    Debug PureLZMA_Archive_GetArchiveInfo(@ArchiveInfo) 
    Debug "----------------" 
    Debug "CRCUnpacked    = $" + RSet(Hex(ArchiveInfo\CRCUnpacked, #PB_Long), 8, "0") 
    Debug "CRCPacked      = $" + RSet(Hex(ArchiveInfo\CRCPacked, #PB_Long), 8, "0") 
    Debug "CRCPacked CALC = $" + RSet(Hex(PureLZMA_Archive_GetCRC(), #PB_Long), 8, "0") 
    Debug "LengthUnpacked = " + Str(ArchiveInfo\LengthUnpacked) 
    Debug "LengthPacked   = " + Str(ArchiveInfo\LengthPacked) + " (" + StrF(ArchiveInfo\LengthPacked / ArchiveInfo\LengthUnpacked * 100, 2) + "% of original)" 
    Debug "LZMAFlags      = " + Str(ArchiveInfo\LZMAFlags) 
    Debug "FileAttributes = $" + Hex(ArchiveInfo\FileAttributes) 
    Debug "Filename       = '" + ArchiveInfo\FileName + "'" 
    ; 
    PureLZMA_Archive_Extract(#PB_Compiler_Home + "\Examples\PureLZMA\" + ArchiveInfo\FileName) 
    ; 
    While PureLZMA_Archive_FindNext() 
      ; 
      Debug PureLZMA_Archive_GetArchiveInfo(@ArchiveInfo) 
      Debug "----------------" 
      Debug "CRCUnpacked    = $" + RSet(Hex(ArchiveInfo\CRCUnpacked, #PB_Long), 8, "0") 
      Debug "CRCPacked      = $" + RSet(Hex(ArchiveInfo\CRCPacked, #PB_Long), 8, "0") 
      Debug "CRCPacked CALC = $" + RSet(Hex(PureLZMA_Archive_GetCRC(), #PB_Long), 8, "0") 
      Debug "LengthUnpacked = " + Str(ArchiveInfo\LengthUnpacked) 
      Debug "LengthPacked   = " + Str(ArchiveInfo\LengthPacked) + " (" + StrF(ArchiveInfo\LengthPacked / ArchiveInfo\LengthUnpacked * 100, 2) + "% of original)" 
      Debug "LZMAFlags      = " + Str(ArchiveInfo\LZMAFlags) 
      Debug "FileAttributes = $" + Hex(ArchiveInfo\FileAttributes) 
      Debug "Filename       = '" + ArchiveInfo\FileName + "'" 
      ; 
      PureLZMA_Archive_Extract(#PB_Compiler_Home + "\Examples\PureLZMA\" + ArchiveInfo\FileName) 
      ; 
    Wend 
  EndIf 
  ; 
  PureLZMA_Archive_Close() 
EndIf
Also, some code snipet showing the problem would help.

Posted: Tue Jan 13, 2009 1:53 pm
by graves
That's my code (resumed).
I noticed a difference with yours: To compress I'm using KeepPath = 1.

Code: Select all

Global option.s, filenam.s, ArchiveInfo.LZMA_ArchiveInfo

Procedure CompressRecursive(fldname.s)
  Protected nDir.l, pthfldr.s, entryname.s
  pthfldr = fldname+"\"

  nDir = ExamineDirectory(#PB_Any, fldname, "*.*")
  While NextDirectoryEntry(nDir)
    entryname = DirectoryEntryName(nDir)
    if entryname <> "." AND entryname <> ".."
      entryname = pthfldr+entryname
      select DirectoryEntryType(nDir)
        case #PB_DirectoryEntry_Directory: CompressRecursive(entryname)
        case #PB_DirectoryEntry_File     : PureLZMA_Archive_Compress(entryname, 1)
      endselect
    endif
  Wend
  FinishDirectory(nDir)
EndProcedure

Procedure Display_File()
  PureLZMA_Archive_GetArchiveInfo(@ArchiveInfo) 
  Debug "----------------" 
  Debug "CRCUnpacked    = $" + RSet(Hex(ArchiveInfo\CRCUnpacked, #PB_Long), 8, "0") 
  Debug "CRCPacked      = $" + RSet(Hex(ArchiveInfo\CRCPacked, #PB_Long), 8, "0") 
  Debug "CRCPacked CALC = $" + RSet(Hex(PureLZMA_Archive_GetCRC(), #PB_Long), 8, "0") 
  Debug "LengthUnpacked = " + Str(ArchiveInfo\LengthUnpacked) 
  Debug "LengthPacked   = " + Str(ArchiveInfo\LengthPacked) + " (" + StrF(ArchiveInfo\LengthPacked / ArchiveInfo\LengthUnpacked * 100, 2) + "% of original)" 
  Debug "LZMAFlags      = " + Str(ArchiveInfo\LZMAFlags) 
  Debug "FileAttributes = $" + Hex(ArchiveInfo\FileAttributes) 
  Debug "Filename       = '" + ArchiveInfo\Filename + "'" 
EndProcedure


; MAIN LINE

option.s = ProgramParameter()
filenam.s = ProgramParameter()

     select option
       case "-c"
        fileTCz = StringField(GetFilePart(filenam),1,".")+".arc"
        if PureLZMA_Archive_Create(fileTCz)
          if FileSize(filenam) = -2: CompressRecursive(filenam)
          else: PureLZMA_Archive_Compress(filenam, 1): endif
          PureLZMA_Archive_Close()
        endif

      case "-v":
        if PureLZMA_Archive_Read(filenam)
          if PureLZMA_Archive_FindFirst()
            Display_File()
            while PureLZMA_Archive_FindNext()
              Display_File()
            wend
          endif
          PureLZMA_Archive_Close()
        else
          Messagerequester("error","Could not open the file")
        endif
    endselect

END

Posted: Tue Jan 13, 2009 2:46 pm
by gnozal
This works here (archive created, content listed):

Code: Select all

Global option.s, filenam.s, ArchiveInfo.LZMA_ArchiveInfo 

Procedure CompressRecursive(fldname.s) 
  Protected nDir.l, pthfldr.s, entryname.s 
  pthfldr = fldname+"\" 
  
  nDir = ExamineDirectory(#PB_Any, fldname, "*.*") 
  While NextDirectoryEntry(nDir) 
    entryname = DirectoryEntryName(nDir) 
    If entryname <> "." And entryname <> ".." 
      entryname = pthfldr+entryname 
      Select DirectoryEntryType(nDir) 
        Case #PB_DirectoryEntry_Directory: CompressRecursive(entryname) 
        Case #PB_DirectoryEntry_File     :
          PureLZMA_Archive_Compress(entryname, 1) 
      EndSelect 
    EndIf 
  Wend 
  FinishDirectory(nDir) 
EndProcedure 

Procedure Display_File() 
  PureLZMA_Archive_GetArchiveInfo(@ArchiveInfo) 
  Debug "----------------" 
  Debug "CRCUnpacked    = $" + RSet(Hex(ArchiveInfo\CRCUnpacked, #PB_Long), 8, "0") 
  Debug "CRCPacked      = $" + RSet(Hex(ArchiveInfo\CRCPacked, #PB_Long), 8, "0") 
  Debug "CRCPacked CALC = $" + RSet(Hex(PureLZMA_Archive_GetCRC(), #PB_Long), 8, "0") 
  Debug "LengthUnpacked = " + Str(ArchiveInfo\LengthUnpacked) 
  Debug "LengthPacked   = " + Str(ArchiveInfo\LengthPacked) + " (" + StrF(ArchiveInfo\LengthPacked / ArchiveInfo\LengthUnpacked * 100, 2) + "% of original)" 
  Debug "LZMAFlags      = " + Str(ArchiveInfo\LZMAFlags) 
  Debug "FileAttributes = $" + Hex(ArchiveInfo\FileAttributes) 
  Debug "Filename       = '" + ArchiveInfo\FileName + "'" 
EndProcedure 


; MAIN LINE 


; <------------------------------------

; 1st : create archive
option.s = "-c"
filenam.s = #PB_Compiler_Home + "Examples\PureLZMA"

; 2nd : read archive
; option.s = "-v"
; filenam.s = #PB_Compiler_Home + "program\PureLZMA.arc"


Select option 
  Case "-c" 
    fileTCz.s = StringField(GetFilePart(filenam),1,".")+".arc" 
    If PureLZMA_Archive_Create(fileTCz) 
      Debug "Create " + fileTCz
      If FileSize(filenam) = -2: CompressRecursive(filenam) 
      Else
        PureLZMA_Archive_Compress(filenam, 1)
      EndIf 
      PureLZMA_Archive_Close() 
    EndIf 
    
  Case "-v": 
    If PureLZMA_Archive_Read(filenam) 
      If PureLZMA_Archive_FindFirst() 
        Display_File() 
        While PureLZMA_Archive_FindNext() 
          Display_File() 
        Wend 
      EndIf 
      PureLZMA_Archive_Close() 
    Else 
      MessageRequester("error","Could not open the file") 
    EndIf 
EndSelect 

End
Run one time to create archive ; uncomment remarks ; run a 2nd time to view the archive content. You may need to adjust the archive path.
The only issue I found is PureLZMA_Archive_Compress() doesn't return a correct value. I just fixed it. Download updated.

Posted: Tue Jan 13, 2009 3:47 pm
by graves
Thanks, gnozal,
Your code is very similar to mine. (in fact, they seems equals).

I must have any other bug on my program. Thanks, again, for testing.

Posted: Tue Jan 13, 2009 3:59 pm
by gnozal
graves wrote:Your code is very similar to mine. (in fact, they seems equals).
Yes, I used your code, I just changed the program parameters.

Be aware that to update an archive (add files to an existing archive), you have to open it with PureLZMA_Archive_Update() and not PureLZMA_Archive_Create().

Code: Select all

; Create new archive
If PureLZMA_Archive_Create("c:\purebasic430\program\TestArchive.arc")
  PureLZMA_Archive_Compress("c:\PureBasic430\Program\PureLZMA_Test_1.pb", 1)
  PureLZMA_Archive_Close()
EndIf
; Update existing archive
If PureLZMA_Archive_Update("c:\purebasic430\program\TestArchive.arc")
  PureLZMA_Archive_Compress("c:\PureBasic430\Program\PureLZMA_Test_2.pb", 1)
  PureLZMA_Archive_Close()
EndIf

Posted: Thu Jan 15, 2009 4:00 pm
by gnozal
Update

Changes :
- PureLZMA : fixed a possible allocatememory heap corruption when using RC4 encryption
- PureLZMA-Packer : added function PureLZMA_Archive_AddFiles()