http://www.purebasic.com
https://www.purebasic.fr/english/
see sergey's codematalog wrote: Mon Feb 24, 2025 11:07 pmIdle, can I use what you are describing to ungzip a file using PB's own capabilities, without requiring zlib?idle wrote: Sat Jan 11, 2025 5:14 am gzip is just deflate with a header and checksum, why not use deflate?
CompressMemory(*input,len,*output,outputlen,#PB_PackerPlugin_Zip)
Yes, I have got that to work by putting my file in memory, but I would like to not rely on zlib, as I have worked with programs that relied on zlib, and they have had problems with accessing or finding the library.idle wrote: Mon Feb 24, 2025 11:41 pmsee sergey's codematalog wrote: Mon Feb 24, 2025 11:07 pmIdle, can I use what you are describing to ungzip a file using PB's own capabilities, without requiring zlib?idle wrote: Sat Jan 11, 2025 5:14 am gzip is just deflate with a header and checksum, why not use deflate?
CompressMemory(*input,len,*output,outputlen,#PB_PackerPlugin_Zip)
https://www.purebasic.fr/english/viewto ... 21#p633621
it still needs zlib
zlib is linked statically to your executable so there's no problem in finding it.matalog wrote: Tue Feb 25, 2025 12:21 amYes, I have got that to work by putting my file in memory, but I would like to not rely on zlib, as I have worked with programs that relied on zlib, and they have had problems with accessing or finding the library.idle wrote: Mon Feb 24, 2025 11:41 pmsee sergey's codematalog wrote: Mon Feb 24, 2025 11:07 pmIdle, can I use what you are describing to ungzip a file using PB's own capabilities, without requiring zlib?idle wrote: Sat Jan 11, 2025 5:14 am gzip is just deflate with a header and checksum, why not use deflate?
CompressMemory(*input,len,*output,outputlen,#PB_PackerPlugin_Zip)
https://www.purebasic.fr/english/viewto ... 21#p633621
it still needs zlib
I thought that your suggestion would allow a single file to be ungzipped without zlib. Is that not correct?
Code: Select all
UseZipPacker()
ImportC "";zlib.lib"
inflateInit2_.l(*stream.z_stream, windowBits.l, *version, streamsize.l)
inflate.l(*stream.z_stream, flush.l)
inflateEnd.l(*stream.z_stream)
EndImport
Brilliant, so I can compile, and use on another Windows 10 PC without problem? That would be greatzlib is linked statically to your executable so there's no problem in finding it.
if the import is causing issues, say your on linux or osx then you could do the import asNo I didn't mean you could ungzip a file without zlib, gzip is a zlib format.Code: Select all
UseZipPacker() ImportC "";zlib.lib" inflateInit2_.l(*stream.z_stream, windowBits.l, *version, streamsize.l) inflate.l(*stream.z_stream, flush.l) inflateEnd.l(*stream.z_stream) EndImport
.Would this be easy to amend so that instead of ungzip, it will gzip instead?Sergey wrote: Mon Jan 13, 2025 10:37 pm Hi, Rinzwind
I adapted Windows code from this forum to your needs,
just test it and correct lines how you want
Please wait for ending, on my PC it took about 18 sec. buffer 1024
Buffer 1024 * 1024 (1 MB) took 0.1 sec.
No need any PB packer's code like ZIP or LZMA
Code: Select all
EnableExplicit #Z_BUFFER_SIZE = 1024 * 1024 ;- buffer size #ZLIB_VERSION = "1.2.8" #Z_OK = 0 #Z_STREAM_END = 1 #Z_FULL_FLUSH = 3 #Z_FINISH = 4 #ENABLE_GZIP = 16 Structure z_stream Align #PB_Structure_AlignC *next_in.BYTE avail_in.l total_in.l ;uLong *next_out.BYTE avail_out.l total_out.l ;uLong *msg.BYTE *state zalloc.i zfree.i opaque.i data_type.l adler.l ;uLong reserved.l ;uLong ;without this, the inflateInit2() fails with a version error CompilerIf #PB_Compiler_Processor = #PB_Processor_x64 alignment.l CompilerEndIf EndStructure ImportC "zlib.lib" inflateInit2_.l(*stream.z_stream, windowBits.l, *version, streamsize.l) inflate.l(*stream.z_stream, flush.l) inflateEnd.l(*stream.z_stream) EndImport Procedure ungzip(*buf) Protected gzip_strm.z_stream, gzip_opaque.l, *gzip_buffer, *gzip_out, gzip_result.l, gzip_unpacked_size.l Protected buf_memory_size = MemorySize(*buf) Protected *buf2 If buf_memory_size > 0 Debug "gzip_packed_size = " + Str(buf_memory_size) gzip_strm.z_stream gzip_strm\next_in = *buf gzip_strm\avail_in = buf_memory_size gzip_strm\opaque = @gzip_opaque inflateInit2_(gzip_strm, 15 | #ENABLE_GZIP, #ZLIB_VERSION, SizeOf(z_stream)) *gzip_buffer = AllocateMemory(#Z_BUFFER_SIZE) *gzip_out = AllocateMemory(#Z_BUFFER_SIZE) If *gzip_buffer And *gzip_out Repeat gzip_strm\next_out = *gzip_buffer gzip_strm\avail_out = #Z_BUFFER_SIZE gzip_result = inflate(gzip_strm, #Z_FULL_FLUSH) If gzip_result = #Z_OK Or gzip_result = #Z_STREAM_END Or gzip_strm\avail_in = 0 CopyMemory(*gzip_buffer, *gzip_out + MemorySize(*gzip_out) - #Z_BUFFER_SIZE, #Z_BUFFER_SIZE) If gzip_result = #Z_STREAM_END Or gzip_strm\avail_in = 0 Break Else *gzip_out = ReAllocateMemory(*gzip_out, MemorySize(*gzip_out) + #Z_BUFFER_SIZE) EndIf Else If gzip_strm\msg Debug PeekS(gzip_strm\msg, #PB_UTF8) Else Debug "gzip_result = " + Str(gzip_result) EndIf Break EndIf ForEver gzip_unpacked_size = gzip_strm\total_out If gzip_unpacked_size > 0 Debug "gzip_unpacked_size = " + Str(gzip_unpacked_size) Debug "compression ratio: " + StrF(buf_memory_size * 100 / gzip_unpacked_size, 1) + "%" *buf2 = AllocateMemory(gzip_unpacked_size) CopyMemory(*gzip_out, *buf2, gzip_unpacked_size) EndIf buf_memory_size = gzip_unpacked_size FreeMemory(*gzip_out) FreeMemory(*gzip_buffer) inflateEnd(gzip_strm) EndIf Else Debug "buf_memory_size = 0" EndIf ProcedureReturn *buf2 EndProcedure Define t1 = ElapsedMilliseconds() Define url.s = "https://raw.githubusercontent.com/json-iterator/test-data/refs/heads/master/large-file.json" Define NewMap headers.s() headers("Accept-Encoding") = "gzip" Debug url Define req = HTTPRequestMemory(#PB_HTTP_Get, url, 0, 0, 0, headers()) Debug "HTTPRequest Timer: " + Str(ElapsedMilliseconds() - t1) If req Define stat.s = HTTPInfo(req, #PB_HTTP_StatusCode) Define res.s = HTTPInfo(req, #PB_HTTP_Response) Debug Left(res, 100) + "..." Define *buf = HTTPMemory(req) FinishHTTP(req) If *buf Define res2.s = PeekS(*buf, MemorySize(*buf), #PB_UTF8 | #PB_ByteLength) Debug Left(res2, 100) + "..." Define t2 = ElapsedMilliseconds() Define *buf2 = ungzip(*buf) Debug "UnGZIP Timer: " + Str(ElapsedMilliseconds() - t2) If *buf2 res2.s = PeekS(*buf2, MemorySize(*buf2), #PB_UTF8 | #PB_ByteLength) Debug Left(res2, 100) + "..." FreeMemory(*buf2) EndIf FreeMemory(*buf) EndIf EndIf
Code: Select all
UseZipPacker()
UseCRC32Fingerprint()
;- GZip-Header.
Structure uGzipHeader
iId1.b ; ID1, ID2 - magic header (immer 0x1F, 0x8B)
iId2.b
iCm.b ; CM - Compression method DEFLATE.
iFlg.b ; FLG -Bitfield.
iMTime.l; MTIME - Modification timestamp as Unix time.
iXfl.b ; XFL - Additional information specific to the compression method.
iOs.b ; OS - Type of file system or operating system.
EndStructure
Procedure.i GzipMemory(*Data, iDataSize.i, *GzipData.Integer)
; Compresses the passed memory in GZIP format.
; -> *Data: Pointer to the data to be compressed.
; -> iDataSize: The length of the data.
; <- *GzipData: A pointer to the compressed data.
; Return: The length of the compressed data, or zero in the event of an error.
; The memory with the compressed data must be released again.
Protected iReturn.i, fResult.i, iGzipSize.i, iCrc.l, iOffset.i
Protected *uGzipHeader.uGzipHeader
; Check passed parameters.
fResult = Bool(*Data And iDataSize)
If (Not fResult)
Debug("The pointer or the length of the memory is zero. *Data = " + *Data +
", iDataSize = " + iDataSize)
EndIf
If fResult
fResult = Bool(*GzipData)
If (Not fResult)
Debug("The pointer for the return of the compressed data is zero. *GzipData = Null")
EndIf
EndIf
If fResult
*GzipData\i = AllocateMemory(iDataSize + 10000) ; More space if the data cannot be compressed.
If *GzipData\i
iOffset = SizeOf(uGzipHeader) - 2 ; The zlib header is later overwritten with the GZIP header.
iGzipSize = CompressMemory(*Data, iDataSize, *GzipData\i + iOffset, iDataSize + 10000 - iOffset,
#PB_PackerPlugin_Zip, 6)
If iGzipSize
; Write the GZIP header.
*uGzipHeader = *GzipData\i
With *uGzipHeader
\iId1 = $1F
\iId2 = $8B
\iCm = $08
\iFlg = $00
\iMTime = $00 ; Date()
; The zlib magic header (2 bytes) is overwritten here; this is not used.
\iXfl = $04
\iOs = $00
EndWith
; Calculate and write the CRC32 of the uncompressed data.
iCrc = Val("$" + Fingerprint(*Data, iDataSize, #PB_Cipher_CRC32))
; Overwrite the CRC32 (Adler32?) at the end of the compressed data, this is not used.
iOffset = iGzipSize + SizeOf(uGzipHeader) - 2 ; plus Gzip header minus overwritten zlib header.
iOffset - 4 ; Overwrite CRC32 (Adler32?) at the end of the zlib data.
PokeL(*GzipData\i + iOffset, iCrc)
; Write the length of the uncompressed data.
iOffset + 4
PokeL(*GzipData\i + iOffset, iDataSize)
; Return the length of the GZip data.
iReturn = iOffset + 4
EndIf
EndIf
EndIf
ProcedureReturn iReturn
EndProcedure
Code: Select all
Procedure.i UngzipToBuffer(*gzip, gzipBytes.i = #PB_Default) ; #PB_Default --> get gzipBytes from MemorySize(*gzip)
Protected *Buffer = #Null
If (*gzip)
UseZipPacker()
If (gzipBytes = #PB_Default)
gzipBytes = MemorySize(*gzip)
EndIf
If (gzipBytes > 0)
Protected CompressedBytes.i, UncompressedBytes.i, BufferSize.i, UnpackedBytes.i, TempWord.w
If ((PeekA(*gzip + 0) = $1F) And (PeekA(*gzip + 1) = $8B)) ; magic number
If (PeekA(*gzip + 2) = $08) ; DEFLATE compression method
If (PeekA(*gzip + 3) = $00) ; no header flags supported (yet)
UncompressedBytes = PeekL(*gzip + gzipBytes - 4)
If (UncompressedBytes > 0)
CompressedBytes = gzipBytes - (10 + 8)
If (CompressedBytes > 0)
BufferSize = UncompressedBytes + 16
*Buffer = AllocateMemory(BufferSize, #PB_Memory_NoClear)
If (*Buffer)
PokeU(*Buffer + UncompressedBytes - 1, $7F7F)
TempWord = PeekW(*gzip + 8)
PokeA(*gzip + 8, $78) ; zlib magic number
PokeA(*gzip + 9, $9C) ; assume "Default Compression"
UnpackedBytes = UncompressMemory(*gzip + (10 - 2), 2 + CompressedBytes + 4, *Buffer, BufferSize)
PokeW(*gzip + 8, TempWord)
If (UnpackedBytes = -1)
If ((PeekA(*Buffer + UncompressedBytes - 1) <> $7F) And (PeekA(*Buffer + UncompressedBytes) = $7F))
UnpackedBytes = UncompressedBytes
EndIf
EndIf
If (UnpackedBytes = UncompressedBytes)
*Buffer = ReAllocateMemory(*Buffer, UnpackedBytes)
Else
FreeMemory(*Buffer)
*Buffer = #Null
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
ProcedureReturn (*Buffer)
EndProcedure
Code: Select all
Procedure.i UngzipToFile(*gzip, File.s, gzipBytes.i = #PB_Default) ; #PB_Default --> get gzipBytes from MemorySize(*gzip)
Protected Result.i = #False
If (File)
Protected *Buffer = UngzipToBuffer(*gzip, gzipBytes)
If (*Buffer)
Protected FN.i = CreateFile(#PB_Any, File)
If (FN)
Protected BufferSize.i = MemorySize(*Buffer)
If (WriteData(FN, *Buffer, BufferSize) = BufferSize)
Result = #True
EndIf
CloseFile(FN)
EndIf
FreeMemory(*Buffer)
EndIf
EndIf
ProcedureReturn (Result)
EndProcedure