Unclear doc for UncompressMemory, Base64Decoder, and Base64DecoderBuffer

[nsstudios]

OutputSize
The memory buffer size to store the uncompressed data. It must be at least of the size of the uncompressed data.

Seems like a copy paste from CompressMemory, but what is the actual remark? Is it OK if it's the same size as compressed data, or does it need to be bigger, and by how much? If my buffer is too big, will the packer downsize it to the proper size?

OutputSize
The size of the output buffer.

The output buffer can be up to 33% smaller than the input buffer, with a minimum size of 64 bytes. It's recommended to get a slightly larger buffer, like 30% smaller to avoid overflows.

Seems like a partially modified copy-paste from Base64EncoderBuffer, but still unclear. Does the output buffer need to be 33% bigger, and what if my buffer is too big, will Decoder downsize it to the proper size?

[BarryG]

I agree. I find the Base64 encoding/decoding commands just way too difficult to understand.

[kenmo]

Hi nsstudios. The help is maybe not clear enough, but it's not nonsensical. You do understand these key facts, right:
- Compressed memory is smaller than the uncompressed original - by an amount that varies depending on the data
- Base64 encoded memory is LARGER than the original - by a fixed amount, a ~4:3 ratio

Seems like a copy paste, but what is the actual remark? Is it OK if it's the same size as compressed data, or does it need to be bigger, and by how much? If it's too big, will the packer resize it to the proper size?

No, it's not OK if the output buffer is the same size as compressed data... The uncompressed output data should always be larger!
No, the packer function will not resize the buffer larger for you. (Can Fred or someone from PB team confirm?)

How big is the uncompressed data? Well currently, that info should be stored with the compressed data, whether it's coming from a file, or another program, or compressed in another part of your program... For example see PackEntrySize() when working with ZIP files.

However, it would be a nice feature request option, for UncompressMemory() to automatically figure out the output size and provide a properly sized buffer back to you!

Seems like a partially modified copy-paste, but still unclear. Does the output buffer need to be 33% bigger, and what if it's too big, will Decoder resize the buffer to the proper size?

No, when decoding Base64 the output buffer does not need to be bigger, it can actually be smaller. The original data takes up LESS bytes than the encoded. It's sort of the opposite of compression.
Again, no, the function will not resize your buffer for you.

If you want to know the exact output buffer size needed, you can try my Base64DecodeBytesNeeded() function here:
https://raw.githubusercontent.com/kenmo ... e64Lib.pbi
In fact, check out that whole includefile, it's my own Base64 library that's much more user friendly than PB's built-in Base64 library, and without the strange quirks like "a minimum size of 64 bytes" -- the Base64 algorithm does not have a minimum number of bytes (well, 1 byte, I guess) so I never understood why PureBasic has this 64-byte buffer requirement.

[nsstudios]

Oh! I think I understand now.
Thank you, Kenmo, so much for explaining and for the include.
I did not understand that I have to keep track of the uncompressed size somewhere if I'm not using the pack itself for storage (memory-only). That makes sense, though.
So my understanding is with pack, store the uncompressed size somewhere, and use the return from CompressMemory as the actual length of the compressed memory, which I can pass to ReallocateMemory to downsize my potentially bigger buffer.
Then use the stored uncompressed size when uncompressing.
With Base64EncoderBuffer, make the buffer original size*1.4, then call Base64EncoderBuffer on it, and pass its return to ReallocateMemory to downsize the buffer to the proper size, then when decoding, make a buffer of the size of the encoded data, use Base64Decoder/DecoderBuffer, and pass its return to ReallocateMemory to downsize the buffer to the proper size.
You're right that it wasn't correct of me to call documentation nonsensical, I'm sorry. I have edited the subject.

Hi nsstudios. The help is maybe not clear enough, but it's not nonsensical. You do understand these key facts, right:
- Compressed memory is smaller than the uncompressed original - by an amount that varies depending on the data
- Base64 encoded memory is LARGER than the original - by a fixed amount, a ~4:3 ratio

Seems like a copy paste, but what is the actual remark? Is it OK if it's the same size as compressed data, or does it need to be bigger, and by how much? If it's too big, will the packer resize it to the proper size?

No, it's not OK if the output buffer is the same size as compressed data... The uncompressed output data should always be larger!
No, the packer function will not resize the buffer larger for you. (Can Fred or someone from PB team confirm?)

How big is the uncompressed data? Well currently, that info should be stored with the compressed data, whether it's coming from a file, or another program, or compressed in another part of your program... For example see PackEntrySize() when working with ZIP files.

However, it would be a nice feature request option, for UncompressMemory() to automatically figure out the output size and provide a properly sized buffer back to you!

Seems like a partially modified copy-paste, but still unclear. Does the output buffer need to be 33% bigger, and what if it's too big, will Decoder resize the buffer to the proper size?

No, when decoding Base64 the output buffer does not need to be bigger, it can actually be smaller. The original data takes up LESS bytes than the encoded. It's sort of the opposite of compression.
Again, no, the function will not resize your buffer for you.

If you want to know the exact output buffer size needed, you can try my Base64DecodeBytesNeeded() function here:
https://raw.githubusercontent.com/kenmo ... e64Lib.pbi
In fact, check out that whole includefile, it's my own Base64 library that's much more user friendly than PB's built-in Base64 library, and without the strange quirks like "a minimum size of 64 bytes" -- the Base64 algorithm does not have a minimum number of bytes (well, 1 byte, I guess) so I never understood why PureBasic has this 64-byte buffer requirement.

[Andre]

Any concrete suggestions what should added/changed in the docs?

(Maybe it's a task for Fred.... I'm not familiar with using Packer / Cipher library, sorry)

[nsstudios]

(edit with new formulas)
Yes. For base64EncoderBuffer:

If the padding was used (#PB_Cipher_NoPadding flag wasn't specified):

Code: Select all

OutputSize=((InputSize+2)/3)<<2

If the padding wasn't used (#PB_Cipher_NoPadding flag was specified):

Code: Select all

OutputSize=((InputSize<<2)|2)/3

If OutputSize is less than 64, make it 64 to fit the minimum size requirement.
OutputSize can now be used to allocate the *OutputBuffer and store the encoded message.

For base64decoder/base64decoderBuffer:

1. If the last two bytes/ascii characters in input are "==", set padding to 2.
2. If only the last byte/ascii character in input is "=", set padding to 1.
3. Otherwise padding is 0.

Code: Select all

OutputSize=((InputSize*3)/4)-padding

If OutputSize is smaller than 64, make it 64 to fit the minimum size requirement.
OutputSize can now be used to allocate *OutputBuffer, and store the decoded message.

For UncompressMemory and CompressMemory, maybe just add a remark that the uncompressed/original size needs to be stored manually and used for UncompressMemory, unless one is using pack, which currently can't be read from memory.
I welcome any suggestions, I've tested things with these algos and they seem to work.

Any concrete suggestions what should added/changed in the docs?

(Maybe it's a task for Fred.... I'm not familiar with using Packer / Cipher library, sorry)

[Andre]

Thank you for this suggestions!
Including formulas and/or an alternative description is something Fred should decide. So we wait for a reply from him