Getting the size of a buffer that can (optionally) be a string

[Quin]

Hi,
In my app, I have an encryption/decryption method that makes PB's AES encoder/decoder much more user-friendly. In the code, I use MemorySize() to get the size to pass. This works in most cases, until I'm trying to encrypt a raw string, like:

Code: Select all

QEncrypt(@JSON$, *Output)

It seems that MemorySize() in this case is telling me how many bytes the string takes up in memory. Quite helpful actually, just not for what I'm trying to do (I'm not trying to encrypt all 942000 bytes, at least I don't think so). I think MemoryStringLength() is more what I want in this case. However, I want it to still be general-purpose. Is there a way for me to dynamically tell if the memory contains a string and only call MemoryStringLength() then, or maybe a function that just handles this for me?
Thanks!

[STARGÅTE]

MemoryStringLength() gives the length of characters, which is not helpful when trying encryption and decryption.
You need the byte length.
MemorySize(), however, is only for an allocated memory, not for a string buffer received by the @ operator.
StringByteLength() gives the size of the buffer needed by the string in the respective format. Here you have to pass the string as it is.

Actually, I'm not sure how QEncrypt(@JSON$, *Output) should work in general (non-string cases), when you not pass the length as well.
As I mentioned, you can not use MemorySize(), if the input/output is not allocated by AllocateMemory().

[Quin]

The function is declared like:

Code: Select all

Declare QEncrypt(*Input, *Output)
Declare QDecrypt(*Input, *Output)

I just call MemorySize(*Output) and pass that as the Size parameter. Passing the string does not seem to like that, though. How would you propose I encrypt this string then?

MemoryStringLength() gives the length of characters, which is not helpful when trying encryption and decryption.
You need the byte length.
MemorySize(), however, is only for an allocated memory, not for a string buffer received by the @ operator.
StringByteLength() gives the size of the buffer needed by the string in the respective format. Here you have to pass the string as it is.

Actually, I'm not sure how QEncrypt(@JSON$, *Output) should work in general (non-string cases), when you not pass the length as well.
As I mentioned, you can not use MemorySize(), if the input/output is not allocated by AllocateMemory().

[STARGÅTE]

I just call MemorySize(*Output) and pass that as the Size parameter.

Which only works, if *Output is an allocated memory with AllocateMemory().

Passing the string does not seem to like that, though.

But you did not pass the string into *Output. You do it for the input.
The length of a string in PureBasic is two times it length in characters, because the string is stored in unicode format (2 byte each character).
The allocated output buffer have to be then allocated with AllocateMemory(StringByteLength(String, #PB_Unicode)).
But there is not only one solution. You can handle the string also in ASCII or UTF8 before encryption, but you can never store the encrypted string in an other string because of its binary data from after encryption!

[Quin]

This is what I do. This is for anetworking application, and I want to send the length of the packet then the encrypted data. I have a macro called FinishPacket that handles this for all my packet types:

Code: Select all

Macro FinishPacket
  JSON$ = ComposeJSON(JSONID)
  Length = StringByteLength(JSON$)
  *Buffer = AllocateMemory(4 + Length)
  *Enc = AllocateMemory(Length)
  PokeL(*Buffer, Length)
  QEncrypt(@JSON$, *Enc)
  CopyMemory(*Enc, *Buffer + 4, Length)
  SendNetworkData(ClientID, *Buffer, 4 + Length)
  FreeJSON(JSONID)
  FreeMemory(*Buffer)
  FreeMemory(*Enc)
EndMacro

On the server, I then:

Code: Select all

        Define *LengthBuffer = AllocateMemory(4)
        ReceiveNetworkData(ClientID, *LengthBuffer, 4)
        Define Length = PeekL(*LengthBuffer)
        FreeMemory(*LengthBuffer)
        If Length <= 0 Or Length >= 65535
          Continue
        EndIf
        Define *OrigBuffer = AllocateMemory(Length)
        Define *DecBuffer = AllocateMemory(Length)
        ReceiveNetworkData(ClientID, *OrigBuffer, Length)
        QDecrypt(*OrigBuffer, *DecBuffer)
        Define Packet$ = PeekS(*DecBuffer, Length, #PB_UTF8)
        FreeMemory(*OrigBuffer)
        FreeMemory(*DecBuffer)
        ParseMessage(ClientID, Packet$)

It gets the length just fine, but can't seem to decrypt the string. I only get the first character in Packet$. I assumed it was some issue with buffer size, but I'm honestly not sure, this is the first project I've ever truly used encryption in. I know the solution is probably laughably simple, but I just can't find it.

I just call MemorySize(*Output) and pass that as the Size parameter.

Which only works, if *Output is an allocated memory with AllocateMemory().

Passing the string does not seem to like that, though.

But you did not pass the string into *Output. You do it for the input.
The length of a string in PureBasic is two times it length in characters, because the string is stored in unicode format (2 byte each character).
The allocated output buffer have to be then allocated with AllocateMemory(StringByteLength(String, #PB_Unicode)).
But there is not only one solution. You can handle the string also in ASCII or UTF8 before encryption, but you can never store the encrypted string in an other string because of its binary data from after encryption!

[STARGÅTE]

Because you try to read the string as UTF8: PeekS(*DecBuffer, Length, #PB_UTF8), but it was originally #PB_Unicode: @JSON$.

Try PeekS(*DecBuffer, Length, #PB_Unicode). Probaby you have to divide the length by 2.

[Quin]

Yup, that did it! Switching to #PB_Unicode in the PeekS() call and doing Length / 2 worked perfectly, thanks loads!
Just curious, is there a reason JSON can't be #PB_UTF8?

[infratec]

Strings inside PB are always Unicode

[Quin]

Strings inside PB are always Unicode

How is unicode different from UTF-8? I thought UTF-8 is the default for most unicode operations?

[STARGÅTE]

Strings inside PB are always Unicode

How is unicode different from UTF-8? I thought UTF-8 is the default for most unicode operations?

The name is a little bit misleading.
PureBasic can handle the basic multilingual plane (BMP) of the Unicode-Standard.
This characters are stored internally always as UTF-16 (in PureBasic called #PB_Unicode). The full Unicode-Standard can be stored in UFT-32 or UTF-8, where the latter has a variable character-byte-length and is therefore not suitable for internal use, but for data transfer.
For example, if you use WriteString() the string is stored in UTF-8 encoding.
JSON itself is UTF-8 encoded, but at the moment you transfer it into PureBasic, it is encoded as UTF-16 internally.
You can use UTF8(JSON$) instead of @JSON$, to receive an allocated memory of a string in UTF-8 format.

[Quin]

That makes sense, thanks!

[NicTheQuick]

Also remember that there are different types of memory that can be used:

Code: Select all

Define fixedString.s{11} = "fixedString"
Define dynamicString.s = "dynamicString"

; Memory in Stack (not allocated dynamically)
Debug @fixedString
Debug @"constantString"

; Memory in Heap (allocated dynamically)
Debug @dynamicString

If I remember it right `MemorySize()` can only be used with memory that was allocated in the Heap anyways.