PB6.11 Need help with Un/CompressMemory and Base64En/Decode

[dige]

To transfer large amounts of data from a web client to a web server / CGI, I need string compression and Base64 encoding.

The following code works with small texts, but > 100 characters something goes wrong. Base64Decode then only returns -1

I'm too stupid - I'm missing something - but what?

Code: Select all

UseZipPacker()

Procedure.s UnCompressStringBase64 (txt.s)
  Protected *out, size, unpacksize, *mem, Result.s, base64size

  size = StringByteLength(txt)
  
  If size
    size * 1.5
    *out = AllocateMemory(size)
  EndIf

  If *out
  
    base64size = Base64Decoder(txt, *out, MemorySize(*out))
    
    If base64size > 0
      *mem = AllocateMemory(base64size * 2)
      If *mem
        unpacksize = UncompressMemory(*out, base64size, *mem, MemorySize(*mem), #PB_PackerPlugin_Zip)
      EndIf
    EndIf  
    
    FreeMemory(*out)
    
    If unpacksize > 0
      Result = PeekS(*mem, unpacksize, #PB_Ascii)
    EndIf
    
    FreeMemory(*mem)
  EndIf
  
  Debug "Uncompressed: " + Result
  
  ProcedureReturn Result
EndProcedure

Procedure.s CompressStringBase64 (txt.s)
  Protected Result.s
  
  size = StringByteLength(txt)
  *mem = Ascii(txt)
  
  If *mem
    *out = AllocateMemory(MemorySize(*mem) * 1.5) 
  EndIf
  
  If *out
    compressedSize = CompressMemory(*mem, MemorySize(*mem), *out, MemorySize(*out), #PB_PackerPlugin_Zip)
  EndIf
  
  FreeMemory(*mem)
  
  If compressedSize > 0
    Result = Base64Encoder(*out, compressedSize, #PB_Cipher_URL)
  EndIf
  
  Debug "Compressed: " + Result
  ProcedureReturn Result
EndProcedure


; Please try with Space(10) and Space(100)
; does it work allways for you?

Debug UnCompressStringBase64(CompressStringBase64( "[ A () C()" + Space(20) + "D$%& G ]"))

[RASHAD]

Try

Code: Select all

   If base64size > 0
      *mem = AllocateMemory(base64size*Len(txt))
      If *mem
        unpacksize = UncompressMemory(*out, base64size, *mem, MemorySize(*mem), #PB_PackerPlugin_Zip)
      EndIf
    EndIf  

[infratec]

Main fault:

Code: Select all

If base64size > 0
      *mem = AllocateMemory(base64size * 10)
      If *mem
        unpacksize = UncompressMemory(*out, base64size, *mem, MemorySize(*mem), #PB_PackerPlugin_Zip)
      EndIf
    EndIf  

If you zip many continious spaces, the size is not only reduced by factor 2. The factor is much larger.
I used above 10 instead of 2, but maybe this is still to small.
It works with Space(200).

[infratec]

Same 'bug' on the other side.

If you compress 5 bytes, the output length is larger, because there is also a zip header.

And your FreeMemory() calls are not correct placed.

A modified version:

Code: Select all

Procedure.s UnCompressStringBase64(txt.s)
  
  Protected *out, size, unpacksize, *mem, Result.s, base64size
  
  
  size = Len(txt)
  If size
    
    If size < 64
      size = 64
    EndIf
    
    *out = AllocateMemory(size)
    If *out
      
      base64size = Base64Decoder(txt, *out, MemorySize(*out))
      If base64size > 0
        
        *mem = AllocateMemory(base64size * 10)
        If *mem
          unpacksize = UncompressMemory(*out, base64size, *mem, MemorySize(*mem), #PB_PackerPlugin_Zip)
          If unpacksize > 0
            Result = PeekS(*mem, unpacksize, #PB_Ascii)
          EndIf
      
          FreeMemory(*mem)
        EndIf
      EndIf  
      
      FreeMemory(*out)
      
    EndIf
  EndIf
  
  Debug "Uncompressed: " + Result
  
  ProcedureReturn Result
  
EndProcedure


Procedure.s CompressStringBase64 (txt.s)
  
  Protected Result.s, Size.i
  
  
  *mem = Ascii(txt)
  If *mem
    
    Size = MemorySize(*mem)
    If Size < 100
      Size = 100
    EndIf
    
    *out = AllocateMemory(Size) 
    If *out
      compressedSize = CompressMemory(*mem, MemorySize(*mem), *out, MemorySize(*out), #PB_PackerPlugin_Zip)
      
      If compressedSize > 0
        Result = Base64Encoder(*out, compressedSize, #PB_Cipher_URL)
      EndIf
      
      FreeMemory(*out)
      
    EndIf
    
    FreeMemory(*mem)
    
  EndIf
  
  Debug "Compressed: " + Result
  
  ProcedureReturn Result
  
EndProcedure



; Please try with Space(10) and Space(100)
; does it work allways for you?

UseZipPacker()

Debug UnCompressStringBase64(CompressStringBase64( "[ A () C()" + Space(2) + "D$%& G ]"))

[dige]

Thank you guys for your help. I am very happy about it. I wasted hours troubleshooting and would never have thought that the memory allocation was insufficient. Many, many thanks for the help!

I think out loud:
So if the unpacking size can increase dramatically, you might even have to set a minimum size independent of the size calculation or even save the original size and use it when unpacking?

[infratec]

If you calc the compression rate from this link:

https://linux.101hacks.com/archive-comp ... p-command/

you will see a factor > 16

In my opinion it would be really the best thing if you add the uncompressed size.
Maybe as a fixed 8 byte (quad) hex string in front of the Base64 string like:

Code: Select all

Size.q = 1
Send$ = RSet(Hex(Size), 16, "0") + "ABCDEF=="
Debug Send$

[dige]

Ok, this is my new attempt. Now it also seems to work fine, also with bigger sizes..
Thanks again for your help.

Code: Select all

; String compression + Base64 coding - for network data transmission of texts

UseZipPacker()

Procedure.s UnCompressStringBase64(txt.s)
  Protected *out, size, unpacksize, *mem, Result.s, base64size, origSize
  
  If Mid(txt, 1, 2) = "/+"
    If Mid(txt, 11, 2) = "+/"
      origSize = Val("$" + Mid(txt, 3, 8))
      txt = Mid(txt, 13)
    EndIf
  EndIf
  
  size = StringByteLength(txt)
  
  If size < 64
    size = 64
  EndIf
  
  If size
    *out = AllocateMemory(size)
  EndIf

  If *out
    base64size = Base64Decoder(txt, *out, MemorySize(*out))
    
    If base64size > 0
      
      If origSize = 0
        origSize = base64size * 10 ; may be not enough!
      EndIf
      
      *mem = AllocateMemory(origSize)
      
      If *mem
        unpacksize = UncompressMemory(*out, base64size, *mem, MemorySize(*mem), #PB_PackerPlugin_Zip)
      Else
        FreeMemory(*out)
        ProcedureReturn ""
      EndIf
      
    EndIf  
    
    FreeMemory(*out)
    
    If unpacksize > 0
      Result = PeekS(*mem, unpacksize, #PB_Ascii)
    EndIf
    
    FreeMemory(*mem)
  EndIf
  
  ProcedureReturn Result
EndProcedure

Procedure.s CompressStringBase64(txt.s, addOrigSize = #True)
  Protected Result.s, size, *mem, *out, compressedSize
  
  *mem = Ascii(txt)
  
  size = MemorySize(*mem)
  
  If *mem
    *out = AllocateMemory(size * 2) ; Erhöht für Sicherheit
  Else
    ProcedureReturn ""
  EndIf
  
  If *out
    compressedSize = CompressMemory(*mem, size, *out, MemorySize(*out), #PB_PackerPlugin_Zip)
  Else
    FreeMemory(*mem)
    ProcedureReturn ""
  EndIf
  
  FreeMemory(*mem)
  
  If compressedSize > 0
    Result = Base64Encoder(*out, compressedSize, #PB_Cipher_URL)
  EndIf
  
  FreeMemory(*out)
  
  If Result <> "" And addOrigSize
    Result = "/+" + RSet(Hex(size), 8, "0") + "+/" + Result
  EndIf
  
  ProcedureReturn Result
EndProcedure

txt.s = "ABCD" + Space(10000000) + "XYZ"

Debug Right(UnCompressStringBase64(CompressStringBase64(txt)), 4)