Page 1 of 1
Getting the size of a buffer that can (optionally) be a string
Posted: Tue Jan 23, 2024 5:10 am
by Quin
Hi,
In my app, I have an encryption/decryption method that makes PB's AES encoder/decoder much more user-friendly. In the code, I use MemorySize() to get the size to pass. This works in most cases, until I'm trying to encrypt a raw string, like:
It seems that MemorySize() in this case is telling me how many bytes the string takes up in memory. Quite helpful actually, just not for what I'm trying to do (I'm not trying to encrypt all 942000 bytes, at least I don't think so). I think MemoryStringLength() is more what I want in this case. However, I want it to still be general-purpose. Is there a way for me to dynamically tell if the memory contains a string and only call MemoryStringLength() then, or maybe a function that just handles this for me?
Thanks!
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Tue Jan 23, 2024 1:07 pm
by STARGÅTE
MemoryStringLength() gives the length of characters, which is not helpful when trying encryption and decryption.
You need the byte length.
MemorySize(), however, is only for an allocated memory, not for a string buffer received by the @ operator.
StringByteLength() gives the size of the buffer needed by the string in the respective format. Here you have to pass the string as it is.
Actually, I'm not sure how QEncrypt(@JSON$, *Output) should work in general (non-string cases), when you not pass the length as well.
As I mentioned, you can not use MemorySize(), if the input/output is not allocated by AllocateMemory().
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Tue Jan 23, 2024 3:22 pm
by Quin
The function is declared like:
Code: Select all
Declare QEncrypt(*Input, *Output)
Declare QDecrypt(*Input, *Output)
I just call MemorySize(*Output) and pass that as the Size parameter. Passing the string does not seem to like that, though. How would you propose I encrypt this string then?
STARGÅTE wrote: Tue Jan 23, 2024 1:07 pm
MemoryStringLength() gives the length of characters, which is not helpful when trying encryption and decryption.
You need the byte length.
MemorySize(), however, is only for an allocated memory, not for a string buffer received by the @ operator.
StringByteLength() gives the size of the buffer needed by the string in the respective format. Here you have to pass the string as it is.
Actually, I'm not sure how QEncrypt(@JSON$, *Output) should work in general (non-string cases), when you not pass the length as well.
As I mentioned, you can not use MemorySize(), if the input/output is not allocated by AllocateMemory().
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Tue Jan 23, 2024 4:32 pm
by STARGÅTE
Quin wrote: Tue Jan 23, 2024 3:22 pm
I just call MemorySize(*Output) and pass that as the Size parameter.
Which only works, if *Output is an allocated memory with AllocateMemory().
Quin wrote: Tue Jan 23, 2024 3:22 pm
Passing the string does not seem to like that, though.
But you did not pass the string into *Output. You do it for the input.
The length of a string in PureBasic is two times it length in characters, because the string is stored in unicode format (2 byte each character).
The allocated output buffer have to be then allocated with AllocateMemory(StringByteLength(String, #PB_Unicode)).
But there is not only one solution. You can handle the string also in ASCII or UTF8 before encryption, but you can never store the encrypted string in an other string because of its binary data from after encryption!
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Tue Jan 23, 2024 5:19 pm
by Quin
This is what I do. This is for anetworking application, and I want to send the length of the packet then the encrypted data. I have a macro called FinishPacket that handles this for all my packet types:
Code: Select all
Macro FinishPacket
JSON$ = ComposeJSON(JSONID)
Length = StringByteLength(JSON$)
*Buffer = AllocateMemory(4 + Length)
*Enc = AllocateMemory(Length)
PokeL(*Buffer, Length)
QEncrypt(@JSON$, *Enc)
CopyMemory(*Enc, *Buffer + 4, Length)
SendNetworkData(ClientID, *Buffer, 4 + Length)
FreeJSON(JSONID)
FreeMemory(*Buffer)
FreeMemory(*Enc)
EndMacro
On the server, I then:
Code: Select all
Define *LengthBuffer = AllocateMemory(4)
ReceiveNetworkData(ClientID, *LengthBuffer, 4)
Define Length = PeekL(*LengthBuffer)
FreeMemory(*LengthBuffer)
If Length <= 0 Or Length >= 65535
Continue
EndIf
Define *OrigBuffer = AllocateMemory(Length)
Define *DecBuffer = AllocateMemory(Length)
ReceiveNetworkData(ClientID, *OrigBuffer, Length)
QDecrypt(*OrigBuffer, *DecBuffer)
Define Packet$ = PeekS(*DecBuffer, Length, #PB_UTF8)
FreeMemory(*OrigBuffer)
FreeMemory(*DecBuffer)
ParseMessage(ClientID, Packet$)
It gets the length just fine, but can't seem to decrypt the string. I only get the first character in Packet$. I assumed it was some issue with buffer size, but I'm honestly not sure, this is the first project I've ever truly used encryption in. I know the solution is probably laughably simple, but I just can't find it.
STARGÅTE wrote: Tue Jan 23, 2024 4:32 pm
Quin wrote: Tue Jan 23, 2024 3:22 pm
I just call MemorySize(*Output) and pass that as the Size parameter.
Which only works, if *Output is an allocated memory with AllocateMemory().
Quin wrote: Tue Jan 23, 2024 3:22 pm
Passing the string does not seem to like that, though.
But you did not pass the string into *Output. You do it for the input.
The length of a string in PureBasic is two times it length in characters, because the string is stored in unicode format (2 byte each character).
The allocated output buffer have to be then allocated with AllocateMemory(StringByteLength(String, #PB_Unicode)).
But there is not only one solution. You can handle the string also in ASCII or UTF8 before encryption, but you can never store the encrypted string in an other string because of its binary data from after encryption!
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Tue Jan 23, 2024 5:55 pm
by STARGÅTE
Because you try to read the string as UTF8: PeekS(*DecBuffer, Length, #PB_UTF8), but it was originally #PB_Unicode: @JSON$.
Try PeekS(*DecBuffer, Length, #PB_Unicode). Probaby you have to divide the length by 2.
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Tue Jan 23, 2024 8:08 pm
by Quin
Yup, that did it! Switching to #PB_Unicode in the PeekS() call and doing Length / 2 worked perfectly, thanks loads!
Just curious, is there a reason JSON can't be #PB_UTF8?
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Tue Jan 23, 2024 8:14 pm
by infratec
Strings inside PB are always Unicode
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Tue Jan 23, 2024 9:46 pm
by Quin
infratec wrote: Tue Jan 23, 2024 8:14 pm
Strings inside PB are always Unicode
How is unicode different from UTF-8? I thought UTF-8 is the default for most unicode operations?
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Tue Jan 23, 2024 11:25 pm
by STARGÅTE
Quin wrote: Tue Jan 23, 2024 9:46 pm
infratec wrote: Tue Jan 23, 2024 8:14 pm
Strings inside PB are always Unicode
How is unicode different from UTF-8? I thought UTF-8 is the default for most unicode operations?
The name is a little bit misleading.
PureBasic can handle the basic multilingual plane (BMP) of the Unicode-Standard.
This characters are stored internally always as
UTF-16 (in PureBasic called #PB_Unicode). The full Unicode-Standard can be stored in UFT-32 or UTF-8, where the latter has a variable character-byte-length and is therefore not suitable for internal use, but for data transfer.
For example, if you use WriteString() the string is stored in UTF-8 encoding.
JSON itself is UTF-8 encoded, but at the moment you transfer it into PureBasic, it is encoded as UTF-16 internally.
You can use UTF8(JSON$) instead of @JSON$, to receive an allocated memory of a string in UTF-8 format.
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Wed Jan 24, 2024 4:34 am
by Quin
That makes sense, thanks!
Re: Getting the size of a buffer that can (optionally) be a string
Posted: Wed Jan 24, 2024 2:15 pm
by NicTheQuick
Also remember that there are different types of memory that can be used:
Code: Select all
Define fixedString.s{11} = "fixedString"
Define dynamicString.s = "dynamicString"
; Memory in Stack (not allocated dynamically)
Debug @fixedString
Debug @"constantString"
; Memory in Heap (allocated dynamically)
Debug @dynamicString
If I remember it right `MemorySize()` can only be used with memory that was allocated in the Heap anyways.