How to compress with zlib

moob · Post by **moob** » Wed Oct 16, 2013 6:39 am

Hi...

I need some help.

I need open an dds file and compress with zlib and save compressed.

Please any help will be great.

Thanks.

JHPJHP · Post by **JHPJHP** » Wed Oct 16, 2013 7:06 am

Check out this thread: http://www.purebasic.fr/english/viewtop ... 13&t=56969

My last post... open your dds file with ReadFile... WriteData.

moob · Post by **moob** » Wed Oct 16, 2013 7:15 am

JHPJHP wrote:Check out this thread: http://www.purebasic.fr/english/viewtop ... 13&t=56969

My last post... open your dds file with ReadData... WriteData.

thanks but I have test this and I use PB 5.00 and give me an error

JHPJHP · Post by **JHPJHP** » Wed Oct 16, 2013 7:17 am

There is a dll version (decompression) near the top of the thread that can be modified for compression, but if you want to post some code - I'll have a look... I'm on PB 5.20 though; not sure if it's using the same zlib library?

moob · Post by **moob** » Thu Oct 17, 2013 3:48 am

I have a problem with this.

Code: Select all

    
    OpenLibrary(0, "zlib1.dll")
    file$ = "temp\teste.dds"
    ReadFile(0, file$)
    SourceLen = Lof(0)
    *SourceMem= AllocateMemory(SourceLen) 
    ReadData(0, *SourceMem,SourceLen)
    CloseFile(0)
    
    ;calculate buffer, allocate memory and compress
    DestLen = ((Sourcelen +(((Sourcelen)/1000)+1)+12))
    *DestMem=AllocateMemory(DestLen)
    result = CallCFunction(0, "compress", *DestMem, @DestLen, *SourceMem, SourceLen)
    
    ;write compressed memory to file
    If result = 0
      CreateFile(1, "temp\teste2.dds")
      WriteData(1, *DestMem, DestLen+1)
      CloseFile(1)
    EndIf
    CloseLibrary(0)

the file "teste.dds" have 2MB, when compressed the size is 56kb, but if I uncompress the file compressed the size is the same 56kb and not 2MB

what I do wrong?

skywalk · Post by **skywalk** » Thu Oct 17, 2013 4:37 am

I use ImportC and "_compress2".

Code: Select all

ImportC "zlib.lib"
  zl_version.i()                                                  As "_zlibVersion"
  zl_compress2.i(*mTo_i, *lenTo_i, *mFrom_i, lenFrom.i, Level.i)  As "_compress2"
  zl_uncompress.i(*mTo_i, *lenTo_i, *mFrom_i, lenFrom.i)          As "_uncompress"
  zl_crc32.i(crc32.i, *buf_i, len.i)                              As "_crc32"
  zl_compressBound.i(lenFrom.i)                                   As "_compressBound" ; Bugs fixed in v1.2.4
EndImport

; compress snippet...
  Level = #ZL_BEST_COMPRESSION ;#ZL_DEFAULT_COMPRESSION
  If *mFrom
    If lenFrom = #PB_Default
      lenFrom = MemorySize(*mFrom)
    EndIf
    ; LenTo >= 0.1% of LenFrom + 12 bytes
    Protected.i lenTo = zl_compressBound(lenFrom) ; Round(lenFrom * 1.001,#PB_Round_Up) + 12
    *mTo = AllocateMemory(lenTo)
    If *mTo
      *p\err = zl_compress2(*mTo, @lenTo, *mFrom, lenFrom, Level)
      If *p\err = #ZL_OK
        *mTo = ReAllocateMemory(*mTo, lenTo)
      Else
        FreeMemory(*mTo)
        *mTo = 0
      EndIf
    EndIf
  EndIf
  ; *mTo = pointer to compressed memory

; uncompress snippet...
  ; Return lenTo or negative error code.
  Protected.i lenFrom = MemorySize(*mFrom)
  Protected.i lenTo = MemorySize(*mTo)
  *p\err = zl_uncompress(*mTo, @lenTo, *mFrom, lenFrom)

moob · Post by **moob** » Thu Oct 17, 2013 4:51 am

skywalk wrote:I use ImportC and "_compress2".

Code: Select all

ImportC "zlib.lib"
  zl_version.i()                                                  As "_zlibVersion"
  zl_compress2.i(*mTo_i, *lenTo_i, *mFrom_i, lenFrom.i, Level.i)  As "_compress2"
  zl_uncompress.i(*mTo_i, *lenTo_i, *mFrom_i, lenFrom.i)          As "_uncompress"
  zl_crc32.i(crc32.i, *buf_i, len.i)                              As "_crc32"
  zl_compressBound.i(lenFrom.i)                                   As "_compressBound" ; Bugs fixed in v1.2.4
EndImport

; compress snippet...
  Level = #ZL_BEST_COMPRESSION ;#ZL_DEFAULT_COMPRESSION
  If *mFrom
    If lenFrom = #PB_Default
      lenFrom = MemorySize(*mFrom)
    EndIf
    ; LenTo >= 0.1% of LenFrom + 12 bytes
    Protected.i lenTo = zl_compressBound(lenFrom) ; Round(lenFrom * 1.001,#PB_Round_Up) + 12
    *mTo = AllocateMemory(lenTo)
    If *mTo
      *p\err = zl_compress2(*mTo, @lenTo, *mFrom, lenFrom, Level)
      If *p\err = #ZL_OK
        *mTo = ReAllocateMemory(*mTo, lenTo)
      Else
        FreeMemory(*mTo)
        *mTo = 0
      EndIf
    EndIf
  EndIf
  ; *mTo = pointer to compressed memory

; uncompress snippet...
  ; Return lenTo or negative error code.
  Protected.i lenFrom = MemorySize(*mFrom)
  Protected.i lenTo = MemorySize(*mTo)
  *p\err = zl_uncompress(*mTo, @lenTo, *mFrom, lenFrom)

Hi...

this example give me an error "#ZL_BEST_COMPRESSION constant not found" I use PB 5.00
------
I have use this code below also and the result is the same with dll

Code: Select all

CompilerSelect #PB_Compiler_OS
CompilerCase #PB_OS_Linux
ImportC #PB_Compiler_Home + "purelibraries/linux/libraries/zlib.a"
CompilerCase #PB_OS_MacOS
  ImportC "/usr/lib/libz.dylib"
  CompilerCase #PB_OS_Windows
    ImportC "zlib.lib"
  CompilerEndSelect
  compress2(*dest,*destlen,*source,sourcelen,Level)
EndImport

    File$ = "temp\teste.dds"
    ReadFile(0, File$)
    sourcelen = Lof(0)
    *SourceMem= AllocateMemory(sourcelen)
    ReadData(0, *SourceMem,sourcelen)
    CloseFile(0)
    
    destLen = ((sourcelen +(((sourcelen)/1000)+1)+12))
    *DestMem=AllocateMemory(destLen)
    Result = compress2(*DestMem,@destLen,*SourceMem,sourcelen,9)
    If Result = 0
    CreateFile(1, "temp\teste2.dds")
    WriteData(1, *DestMem, destLen+1)
    CloseFile(1)
    FreeMemory(*sourceMem)
    FreeMemory(*destMem)
    EndIf

the orginal file have a size 2MB if compress and uncompress the size is 56kb

skywalk · Post by **skywalk** » Thu Oct 17, 2013 5:17 am

Sorry, those are constants from the zlib website. I'm not seeing much wrong with what you're doing, but I am on v5.2. Can you try to simplify the problem by compressing/uncompressing a smaller file?

Code: Select all

#ZL_OK                  = 0
#ZL_STREAM_END          = 1
#ZL_NEED_DICT           = 2
#ZL_ERRNO               = -1
#ZL_STREAM_ERROR        = -2
#ZL_DATA_ERROR          = -3
#ZL_MEM_ERROR           = -4
#ZL_BUF_ERROR           = -5
#ZL_VERSION_ERROR       = -6
#ZL_CRC32_ERROR         = -15
#ZL_NO_COMPRESSION      = 0
#ZL_BEST_SPEED          = 1
#ZL_BEST_COMPRESSION    = 9
#ZL_DEFAULT_COMPRESSION = -1
; Compression level must be #ZL_DEFAULT_COMPRESSION, or between 0 and 9.
; 1 gives best speed, 9 gives best compression, 0 = no compression (input data is simply copied a block at a time).
; #ZL_DEFAULT_COMPRESSION = Level 6, compromise between speed and compression.

moob · Post by **moob** » Thu Oct 17, 2013 6:20 am

skywalk wrote:Sorry, those are constants from the zlib website. I'm not seeing much wrong with what you're doing, but I am on v5.2. Can you try to simplify the problem by compressing/uncompressing a smaller file?

Code: Select all

#ZL_OK                  = 0
#ZL_STREAM_END          = 1
#ZL_NEED_DICT           = 2
#ZL_ERRNO               = -1
#ZL_STREAM_ERROR        = -2
#ZL_DATA_ERROR          = -3
#ZL_MEM_ERROR           = -4
#ZL_BUF_ERROR           = -5
#ZL_VERSION_ERROR       = -6
#ZL_CRC32_ERROR         = -15
#ZL_NO_COMPRESSION      = 0
#ZL_BEST_SPEED          = 1
#ZL_BEST_COMPRESSION    = 9
#ZL_DEFAULT_COMPRESSION = -1
; Compression level must be #ZL_DEFAULT_COMPRESSION, or between 0 and 9.
; 1 gives best speed, 9 gives best compression, 0 = no compression (input data is simply copied a block at a time).
; #ZL_DEFAULT_COMPRESSION = Level 6, compromise between speed and compression.

I use PB 5.00 because I need to use an rwapper of DevIL lib and only found one compatible with PB 5.00.

You can test my code abobe in anything, the result of compression is an corrupt file.

I try uncompress with an external program and the compression generated with my code is corrupt

and I don't now what wrong on the my code abobe to generate an corrupt file.

sorry for my english

JHPJHP · Post by **JHPJHP** » Fri Oct 18, 2013 2:15 am

What error (if any) are you receiving when you run the following?
- uses PureBasic's built-in zlib library (to use the DLL - create Prototypes)
- works with text based files / binary files require some modification to the code
- don't know why, but deflateBound is not doing its job
-- to decrease the size of the deflated file, adjust this line: LengthToWrite = deflateBound(@strm, LengthToRead)
--- to something like: LengthToWrite = deflateBound(@strm, LengthToRead) / 2

Code: Select all

#ENABLE_GZIP = 16
#Z_NULL = 0
#ZLIB_VERSION = "1.2.8"
#Z_FINISH = 4
#Z_DEFAULT_COMPRESSION = -1
#Z_DEFLATED = 8
#MAX_MEM_LEVEL = 9
#Z_DEFAULT_STRATEGY = 0

Structure Z_STREAM Align #PB_Structure_AlignC
  *next_in.Byte
  avail_in.l
  total_in.l
  *next_out.Byte
  avail_out.l
  total_out.l
  *msg.Byte
  *state
  zalloc.l
  zfree.l
  opaque.l
  data_type.i
  adler.l
  reserved.l
  CompilerIf #PB_Compiler_Processor = #PB_Processor_x64
    alignment.l
  CompilerEndIf
EndStructure

ImportC "zlib.lib"
  inflateInit2_(*strm, windowBits.i, Version.s, strm_size)
  inflate(*strm, flush.i)
  inflateEnd(*strm)
  deflateInit2_(*strm, level.i, method.i, windowBits.i, memlevel.i, strategy.i, Version.s, strm_size)
  deflateBound(*strm, sourceLen)
  deflate(*strm, flush.i)
  deflateEnd(*strm)
EndImport

Procedure.s InflatePayload(*Payload, windowBits.i)
  LengthToRead = MemorySize(*Payload)
  LengthToWrite = LengthToRead * 8
  *Output = AllocateMemory(LengthToWrite)
  strm.Z_STREAM
  strm\next_in = *Payload
  strm\avail_in = LengthToRead
  strm\next_out = *Output
  strm\avail_out = LengthToWrite
  strm\zalloc = #Z_NULL
  strm\zfree = #Z_NULL
  strm\opaque = #Z_NULL
  inflateInit2_(@strm, windowBits, #ZLIB_VERSION, SizeOf(Z_STREAM))
  inflate(@strm, #Z_FINISH)
  inflateEnd(@strm)
  sOutput.s = PeekS(*Output, -1, #PB_UTF8)
  FreeMemory(*Output)
  ProcedureReturn sOutput
EndProcedure

Procedure DeflatePayload(sInput.s, windowBits.i)
  LengthToRead = StringByteLength(sInput)
  strm.Z_STREAM
  strm\next_in = @sInput
  strm\avail_in = LengthToRead
  strm\zalloc = #Z_NULL
  strm\zfree = #Z_NULL
  strm\opaque = #Z_NULL
  deflateInit2_(@strm, #Z_DEFAULT_COMPRESSION, #Z_DEFLATED, windowBits, #MAX_MEM_LEVEL, #Z_DEFAULT_STRATEGY, #ZLIB_VERSION, SizeOf(Z_STREAM))
  LengthToWrite = deflateBound(@strm, LengthToRead)
  *Payload = AllocateMemory(LengthToWrite)
  strm\next_out = *Payload
  strm\avail_out = LengthToWrite
  deflate(@strm, #Z_FINISH)
  deflateEnd(@strm)
  ProcedureReturn *Payload
EndProcedure

If ReadFile(0, "Buglist.txt")
  While Eof(0) = 0
    sInput.s + ReadString(0) + #CRLF$
  Wend
  CloseFile(0)
  *Payload = DeflatePayload(sInput, 15 | #ENABLE_GZIP)

  If CreateFile(0, "Buglist.gz")
    WriteData(0, *Payload, MemorySize(*Payload))
    CloseFile(0)
  EndIf
  FreeMemory(*Payload)
EndIf

If OpenFile(0, "Buglist.gz")
  LengthToRead = FileSize("Buglist.gz")
  *Payload = AllocateMemory(LengthToRead)
  ReadData(0, *Payload, LengthToRead)
  CloseFile(0)
  sOutput.s = InflatePayload(*Payload, 15 | #ENABLE_GZIP)
  Debug sOutput

  If CreateFile(0, "Buglist2.txt")
    WriteString(0, sOutput)
    CloseFile(0)
  EndIf
EndIf
FreeMemory(*Payload)

moob · Post by **moob** » Fri Oct 18, 2013 4:04 am

JHPJHP wrote:What error (if any) are you receiving when you run the following?
- uses PureBasic's built-in zlib library (to use the DLL - create Prototypes)
- works with text based files / binary files require some modification to the code
- don't know why, but deflateBound is not doing its job
-- to decrease the size of the deflated file, adjust this line: LengthToWrite = deflateBound(@strm, LengthToRead)
--- to something like: LengthToWrite = deflateBound(@strm, LengthToRead) / 2

Code: Select all

#ENABLE_GZIP = 16
#Z_NULL = 0
#ZLIB_VERSION = "1.2.8"
#Z_FINISH = 4
#Z_DEFAULT_COMPRESSION = -1
#Z_DEFLATED = 8
#MAX_MEM_LEVEL = 9
#Z_DEFAULT_STRATEGY = 0

Structure Z_STREAM Align #PB_Structure_AlignC
  *next_in.Byte
  avail_in.l
  total_in.l
  *next_out.Byte
  avail_out.l
  total_out.l
  *msg.Byte
  *state
  zalloc.l
  zfree.l
  opaque.l
  data_type.i
  adler.l
  reserved.l
  CompilerIf #PB_Compiler_Processor = #PB_Processor_x64
    alignment.l
  CompilerEndIf
EndStructure

ImportC "zlib.lib"
  inflateInit2_(*strm, windowBits.i, Version.s, strm_size)
  inflate(*strm, flush.i)
  inflateEnd(*strm)
  deflateInit2_(*strm, level.i, method.i, windowBits.i, memlevel.i, strategy.i, Version.s, strm_size)
  deflateBound(*strm, sourceLen)
  deflate(*strm, flush.i)
  deflateEnd(*strm)
EndImport

Procedure.s InflatePayload(*Payload, windowBits.i)
  LengthToRead = MemorySize(*Payload)
  LengthToWrite = LengthToRead * 8
  *Output = AllocateMemory(LengthToWrite)
  strm.Z_STREAM
  strm\next_in = *Payload
  strm\avail_in = LengthToRead
  strm\next_out = *Output
  strm\avail_out = LengthToWrite
  strm\zalloc = #Z_NULL
  strm\zfree = #Z_NULL
  strm\opaque = #Z_NULL
  inflateInit2_(@strm, windowBits, #ZLIB_VERSION, SizeOf(Z_STREAM))
  inflate(@strm, #Z_FINISH)
  inflateEnd(@strm)
  sOutput.s = PeekS(*Output, -1, #PB_UTF8)
  FreeMemory(*Output)
  ProcedureReturn sOutput
EndProcedure

Procedure DeflatePayload(sInput.s, windowBits.i)
  LengthToRead = StringByteLength(sInput)
  strm.Z_STREAM
  strm\next_in = @sInput
  strm\avail_in = LengthToRead
  strm\zalloc = #Z_NULL
  strm\zfree = #Z_NULL
  strm\opaque = #Z_NULL
  deflateInit2_(@strm, #Z_DEFAULT_COMPRESSION, #Z_DEFLATED, windowBits, #MAX_MEM_LEVEL, #Z_DEFAULT_STRATEGY, #ZLIB_VERSION, SizeOf(Z_STREAM))
  LengthToWrite = deflateBound(@strm, LengthToRead)
  *Payload = AllocateMemory(LengthToWrite)
  strm\next_out = *Payload
  strm\avail_out = LengthToWrite
  deflate(@strm, #Z_FINISH)
  deflateEnd(@strm)
  ProcedureReturn *Payload
EndProcedure

If ReadFile(0, "Buglist.txt")
  While Eof(0) = 0
    sInput.s + ReadString(0) + #CRLF$
  Wend
  CloseFile(0)
  *Payload = DeflatePayload(sInput, 15 | #ENABLE_GZIP)

  If CreateFile(0, "Buglist.gz")
    WriteData(0, *Payload, MemorySize(*Payload))
    CloseFile(0)
  EndIf
  FreeMemory(*Payload)
EndIf

If OpenFile(0, "Buglist.gz")
  LengthToRead = FileSize("Buglist.gz")
  *Payload = AllocateMemory(LengthToRead)
  ReadData(0, *Payload, LengthToRead)
  CloseFile(0)
  sOutput.s = InflatePayload(*Payload, 15 | #ENABLE_GZIP)
  Debug sOutput

  If CreateFile(0, "Buglist2.txt")
    WriteString(0, sOutput)
    CloseFile(0)
  EndIf
EndIf
FreeMemory(*Payload)

the error I receive when open with an external tool is "File parsing failed, invalid or unsupported file format"

I can't test your code in PB 5.00 because give me an error "Align is not a valid operator"

Can you please adapt your code to work with PB 5.00

thanks

JHPJHP · Post by **JHPJHP** » Fri Oct 18, 2013 6:48 am

That error doesn't make sense (to me) - Align is a part of the Structure syntax?

If your in the 32bit version of PureBasic - change the Z_STREAM structure to this:
- also test with a basic text file and see if you get an error
- what error (if any) do you get now

Code: Select all

Structure Z_STREAM
  *next_in.Byte
  avail_in.l
  total_in.l
  *next_out.Byte
  avail_out.l
  total_out.l
  *msg.Byte
  *state
  zalloc.l
  zfree.l
  opaque.l
  data_type.i
  adler.l
  reserved.l
EndStructure

moob · Post by **moob** » Fri Oct 18, 2013 7:19 am

JHPJHP wrote:That error doesn't make sense (to me) - Align is a part of the Structure syntax?

If your in the 32bit version of PureBasic - change the Z_STREAM structure to this:
- also test with a basic text file and see if you get an error
- what error (if any) do you get now
Code: Select all
Structure Z_STREAM
  *next_in.Byte
  avail_in.l
  total_in.l
  *next_out.Byte
  avail_out.l
  total_out.l
  *msg.Byte
  *state
  zalloc.l
  zfree.l
  opaque.l
  data_type.i
  adler.l
  reserved.l
EndStructure

I write my name "moob" in "Buglist.txt" but in "Buglist2.txt" not appear my name only the first letter.

And I tried with dds file and not working too.

JHPJHP · Post by **JHPJHP** » Fri Oct 18, 2013 8:03 am

Check to see if you have Unicode selected under options - if so remove it... Let me know if that resolves the problem?

moob · Post by **moob** » Fri Oct 18, 2013 5:09 pm

JHPJHP wrote:Check to see if you have Unicode selected under options - if so remove it... Let me know if that resolves the problem?

With txt file work, but with dds file crash.

And I need is for compress dds file with zlib

Here is my dds file for test: http://www9.zippyshare.com/v/80941158/file.html

PureBasic Forums - English

How to compress with zlib

How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib

Re: How to compress with zlib