HttpRequest with gzip

[idle]

gzip is just deflate with a header and checksum, why not use deflate?
CompressMemory(*input,len,*output,outputlen,#PB_PackerPlugin_Zip)

Idle, can I use what you are describing to ungzip a file using PB's own capabilities, without requiring zlib?

see sergey's code
https://www.purebasic.fr/english/viewto ... 21#p633621
it still needs zlib

[matalog]

gzip is just deflate with a header and checksum, why not use deflate?
CompressMemory(*input,len,*output,outputlen,#PB_PackerPlugin_Zip)

Idle, can I use what you are describing to ungzip a file using PB's own capabilities, without requiring zlib?

see sergey's code
https://www.purebasic.fr/english/viewto ... 21#p633621
it still needs zlib

Yes, I have got that to work by putting my file in memory, but I would like to not rely on zlib, as I have worked with programs that relied on zlib, and they have had problems with accessing or finding the library.

I thought that your suggestion would allow a single file to be ungzipped without zlib. Is that not correct?

[idle]

gzip is just deflate with a header and checksum, why not use deflate?
CompressMemory(*input,len,*output,outputlen,#PB_PackerPlugin_Zip)

Idle, can I use what you are describing to ungzip a file using PB's own capabilities, without requiring zlib?

see sergey's code
https://www.purebasic.fr/english/viewto ... 21#p633621
it still needs zlib

Yes, I have got that to work by putting my file in memory, but I would like to not rely on zlib, as I have worked with programs that relied on zlib, and they have had problems with accessing or finding the library.

I thought that your suggestion would allow a single file to be ungzipped without zlib. Is that not correct?

zlib is linked statically to your executable so there's no problem in finding it.
if the import is causing issues, say your on linux or osx then you could do the import as

Code: Select all

UseZipPacker()

ImportC "";zlib.lib"
	inflateInit2_.l(*stream.z_stream, windowBits.l, *version, streamsize.l)
	inflate.l(*stream.z_stream, flush.l)
	inflateEnd.l(*stream.z_stream)
EndImport

No I didn't mean you could ungzip a file without zlib, gzip is a zlib format.

[matalog]

zlib is linked statically to your executable so there's no problem in finding it.
if the import is causing issues, say your on linux or osx then you could do the import as

Code: Select all

UseZipPacker()

ImportC "";zlib.lib"
	inflateInit2_.l(*stream.z_stream, windowBits.l, *version, streamsize.l)
	inflate.l(*stream.z_stream, flush.l)
	inflateEnd.l(*stream.z_stream)
EndImport

No I didn't mean you could ungzip a file without zlib, gzip is a zlib format.

Brilliant, so I can compile, and use on another Windows 10 PC without problem? That would be great .

I'll give it a try.

[matalog]

Hi, Rinzwind
I adapted Windows code from this forum to your needs,
just test it and correct lines how you want
Please wait for ending, on my PC it took about 18 sec. buffer 1024
Buffer 1024 * 1024 (1 MB) took 0.1 sec.
No need any PB packer's code like ZIP or LZMA

Code: Select all

EnableExplicit

#Z_BUFFER_SIZE = 1024 * 1024 ;- buffer size

#ZLIB_VERSION = "1.2.8"

#Z_OK = 0
#Z_STREAM_END = 1
#Z_FULL_FLUSH = 3
#Z_FINISH = 4

#ENABLE_GZIP = 16

Structure z_stream Align #PB_Structure_AlignC
	*next_in.BYTE
	avail_in.l
	total_in.l ;uLong
	
	*next_out.BYTE
	avail_out.l
	total_out.l ;uLong
	
	*msg.BYTE
	*state
	
	zalloc.i
	zfree.i
	opaque.i
	
	data_type.l
	adler.l ;uLong
	reserved.l ;uLong
			   ;without this, the inflateInit2() fails with a version error
	CompilerIf #PB_Compiler_Processor = #PB_Processor_x64
		alignment.l
	CompilerEndIf
EndStructure

ImportC "zlib.lib"
	inflateInit2_.l(*stream.z_stream, windowBits.l, *version, streamsize.l)
	inflate.l(*stream.z_stream, flush.l)
	inflateEnd.l(*stream.z_stream)
EndImport

Procedure ungzip(*buf)	
	Protected gzip_strm.z_stream, gzip_opaque.l, *gzip_buffer, *gzip_out, gzip_result.l, gzip_unpacked_size.l	
	Protected buf_memory_size = MemorySize(*buf)
	Protected *buf2
	
	If buf_memory_size > 0
		Debug "gzip_packed_size = " + Str(buf_memory_size)
		gzip_strm.z_stream
		gzip_strm\next_in = *buf
		gzip_strm\avail_in = buf_memory_size
		gzip_strm\opaque = @gzip_opaque
		
		inflateInit2_(gzip_strm, 15 | #ENABLE_GZIP, #ZLIB_VERSION, SizeOf(z_stream))
		
		*gzip_buffer = AllocateMemory(#Z_BUFFER_SIZE)
		*gzip_out    = AllocateMemory(#Z_BUFFER_SIZE)
		
		If *gzip_buffer And *gzip_out
			Repeat
				gzip_strm\next_out = *gzip_buffer
				gzip_strm\avail_out = #Z_BUFFER_SIZE
				gzip_result = inflate(gzip_strm, #Z_FULL_FLUSH)
				
				If gzip_result = #Z_OK Or gzip_result = #Z_STREAM_END Or gzip_strm\avail_in = 0
					CopyMemory(*gzip_buffer, *gzip_out + MemorySize(*gzip_out) - #Z_BUFFER_SIZE, #Z_BUFFER_SIZE)
					If gzip_result = #Z_STREAM_END Or gzip_strm\avail_in = 0
						Break
					Else
						*gzip_out = ReAllocateMemory(*gzip_out, MemorySize(*gzip_out) + #Z_BUFFER_SIZE)
					EndIf
				Else
					If gzip_strm\msg
						Debug PeekS(gzip_strm\msg, #PB_UTF8)
					Else
						Debug "gzip_result = " + Str(gzip_result)
					EndIf
					Break
				EndIf
			ForEver
			
			gzip_unpacked_size = gzip_strm\total_out
			If gzip_unpacked_size > 0
				Debug "gzip_unpacked_size = " + Str(gzip_unpacked_size)
				Debug "compression ratio: " + StrF(buf_memory_size * 100 / gzip_unpacked_size, 1) + "%"
				
				*buf2 = AllocateMemory(gzip_unpacked_size)
				CopyMemory(*gzip_out, *buf2, gzip_unpacked_size)
			EndIf
			
			buf_memory_size = gzip_unpacked_size
			
			FreeMemory(*gzip_out)
			FreeMemory(*gzip_buffer)
			
			inflateEnd(gzip_strm)
		EndIf
		
	Else
		Debug "buf_memory_size = 0"
	EndIf
	
	ProcedureReturn *buf2
EndProcedure

Define t1 = ElapsedMilliseconds()
Define url.s = "https://raw.githubusercontent.com/json-iterator/test-data/refs/heads/master/large-file.json"
Define NewMap headers.s()
headers("Accept-Encoding") = "gzip"
Debug url
Define req = HTTPRequestMemory(#PB_HTTP_Get, url, 0, 0, 0, headers())
Debug "HTTPRequest Timer: " + Str(ElapsedMilliseconds() - t1)
If req
	Define stat.s = HTTPInfo(req, #PB_HTTP_StatusCode)
	Define res.s = HTTPInfo(req, #PB_HTTP_Response)
	Debug Left(res, 100) + "..."
	Define *buf = HTTPMemory(req)
	FinishHTTP(req)	
	
	If *buf
		Define res2.s = PeekS(*buf, MemorySize(*buf), #PB_UTF8 | #PB_ByteLength)
		Debug Left(res2, 100) + "..."
		
		Define t2 = ElapsedMilliseconds()
		Define *buf2 = ungzip(*buf)
		Debug "UnGZIP Timer: " + Str(ElapsedMilliseconds() - t2)
		
		If *buf2
			res2.s = PeekS(*buf2, MemorySize(*buf2), #PB_UTF8 | #PB_ByteLength)
			Debug Left(res2, 100) + "..."
		
			FreeMemory(*buf2)
		EndIf
		
		FreeMemory(*buf)
	EndIf
EndIf

Would this be easy to amend so that instead of ungzip, it will gzip instead?

[PeDe]

Hello matalog,
here is a procedure that I have tested. A .NET program can read the GZIP-compressed data. The program saves its data in compressed form (.NET GZipStream).

Peter

Code: Select all

UseZipPacker()
UseCRC32Fingerprint()

;- GZip-Header.
Structure uGzipHeader
  iId1.b			; ID1, ID2 - magic header (immer 0x1F, 0x8B)
  iId2.b
  iCm.b		; CM - Compression method DEFLATE.
  iFlg.b  ; FLG -Bitfield.
  iMTime.l; MTIME - Modification timestamp as Unix time.
  iXfl.b  ; XFL - Additional information specific to the compression method.
  iOs.b   ; OS - Type of file system or operating system.
EndStructure

Procedure.i GzipMemory(*Data, iDataSize.i, *GzipData.Integer)
  ; Compresses the passed memory in GZIP format.
  ; -> *Data:			Pointer to the data to be compressed.
  ; -> iDataSize:		The length of the data.
  ; <- *GzipData:	A pointer to the compressed data.
  ; Return: The length of the compressed data, or zero in the event of an error.
  ;	The memory with the compressed data must be released again.
  Protected iReturn.i, fResult.i, iGzipSize.i, iCrc.l, iOffset.i
  Protected *uGzipHeader.uGzipHeader

  ; Check passed parameters.
  fResult = Bool(*Data And iDataSize)
  If (Not fResult)
    Debug("The pointer or the length of the memory is zero. *Data = " + *Data +
      ", iDataSize = " + iDataSize)
  EndIf
  If fResult
    fResult = Bool(*GzipData)
    If (Not fResult)
      Debug("The pointer for the return of the compressed data is zero. *GzipData = Null")
    EndIf
  EndIf
  
  If fResult
    *GzipData\i = AllocateMemory(iDataSize + 10000)	; More space if the data cannot be compressed.
    If *GzipData\i
      iOffset = SizeOf(uGzipHeader) - 2	; The zlib header is later overwritten with the GZIP header.
      iGzipSize = CompressMemory(*Data, iDataSize, *GzipData\i + iOffset, iDataSize + 10000 - iOffset,
        #PB_PackerPlugin_Zip, 6)
      If iGzipSize
        ; Write the GZIP header.
        *uGzipHeader = *GzipData\i
        With *uGzipHeader
          \iId1 = $1F
          \iId2 = $8B
          \iCm = $08
          \iFlg = $00
          \iMTime = $00 ; Date()
          ; The zlib magic header (2 bytes) is overwritten here; this is not used.
          \iXfl = $04
          \iOs = $00
        EndWith
        ; Calculate and write the CRC32 of the uncompressed data.
        iCrc = Val("$" + Fingerprint(*Data, iDataSize, #PB_Cipher_CRC32))
        ; Overwrite the CRC32 (Adler32?) at the end of the compressed data, this is not used.
        iOffset = iGzipSize + SizeOf(uGzipHeader) - 2	; plus Gzip header minus overwritten zlib header.
        iOffset - 4                                   ; Overwrite CRC32 (Adler32?) at the end of the zlib data.
        PokeL(*GzipData\i + iOffset, iCrc)
        ; Write the length of the uncompressed data.
        iOffset + 4
        PokeL(*GzipData\i + iOffset, iDataSize)
        
        ; Return the length of the GZip data.
        iReturn = iOffset + 4
      EndIf		
    EndIf		
  EndIf
  
  ProcedureReturn iReturn
EndProcedure

[kenmo]

I had to take a swing at a single-procedure solution.
Takes a *gzip buffer and returns an unpacked buffer. Uses PB's UseZipPacker() but nothing else.
Does this work for you?

Code: Select all

Procedure.i UngzipToBuffer(*gzip, gzipBytes.i = #PB_Default)  ;  #PB_Default --> get gzipBytes from MemorySize(*gzip)
  Protected *Buffer = #Null
  
  If (*gzip)
    UseZipPacker()
    If (gzipBytes = #PB_Default)
      gzipBytes = MemorySize(*gzip)
    EndIf
    If (gzipBytes > 0)
      Protected CompressedBytes.i, UncompressedBytes.i, BufferSize.i, UnpackedBytes.i, TempWord.w
      
      If ((PeekA(*gzip + 0) = $1F) And (PeekA(*gzip + 1) = $8B)) ; magic number
        If (PeekA(*gzip + 2) = $08) ; DEFLATE compression method
          If (PeekA(*gzip + 3) = $00) ; no header flags supported (yet)
            
            UncompressedBytes = PeekL(*gzip + gzipBytes - 4)
            If (UncompressedBytes > 0)
              CompressedBytes = gzipBytes - (10 + 8)
              If (CompressedBytes > 0)
                BufferSize = UncompressedBytes + 16
                *Buffer = AllocateMemory(BufferSize, #PB_Memory_NoClear)
                If (*Buffer)
                  PokeU(*Buffer + UncompressedBytes - 1, $7F7F)
                  TempWord = PeekW(*gzip + 8)
                  PokeA(*gzip + 8, $78) ; zlib magic number
                  PokeA(*gzip + 9, $9C) ; assume "Default Compression"
                  UnpackedBytes = UncompressMemory(*gzip + (10 - 2), 2 + CompressedBytes + 4, *Buffer, BufferSize)
                  PokeW(*gzip + 8, TempWord)
                  
                  If (UnpackedBytes = -1)
                    If ((PeekA(*Buffer + UncompressedBytes - 1) <> $7F) And (PeekA(*Buffer + UncompressedBytes) = $7F))
                      UnpackedBytes = UncompressedBytes
                    EndIf
                  EndIf
                  
                  If (UnpackedBytes = UncompressedBytes)
                    *Buffer = ReAllocateMemory(*Buffer, UnpackedBytes)
                  Else
                    FreeMemory(*Buffer)
                    *Buffer = #Null
                  EndIf
                EndIf
              EndIf
            EndIf
            
          EndIf
        EndIf
      EndIf
      
    EndIf
  EndIf
  
  ProcedureReturn (*Buffer)
EndProcedure

And a second helper procedure to un-gzip to file:

Code: Select all

Procedure.i UngzipToFile(*gzip, File.s, gzipBytes.i = #PB_Default)  ;  #PB_Default --> get gzipBytes from MemorySize(*gzip)
  Protected Result.i = #False
  If (File)
    Protected *Buffer = UngzipToBuffer(*gzip, gzipBytes)
    If (*Buffer)
      Protected FN.i = CreateFile(#PB_Any, File)
      If (FN)
        Protected BufferSize.i = MemorySize(*Buffer)
        If (WriteData(FN, *Buffer, BufferSize) = BufferSize)
          Result = #True
        EndIf
        CloseFile(FN)
      EndIf
      FreeMemory(*Buffer)
    EndIf
  EndIf
  ProcedureReturn (Result)
EndProcedure

Note, it does everything in-memory rather than iterating chunks of data, so I would not recommend using this for huge/gigabytes of data!

[matalog]

Thanks, I will try those.

[Rinzwind]

Any way to get this into v6.21? PM if needed.
ps forum is extremely slow. (edit: reboot? everything speedy again for now)

[Fred]

We won't add anything to 6.21 but it should be in 6.30.