What is the maximum length a string variable can hold?

[charvista]

PureBasic allows many type variables which can be standard integers, float, double, quad and char numbers or even string characters. Here is the list of the native supported types and a brief description :
...
Name.......Extension...Memory consumption......Range
String........s............ string length + 1...........unlimited
...

In the Help file, under 'Variables', we can read that the length of a String is unlimited.
Then, why does then the following code return an [ERROR] Invalid memory access. (write error at address 39788544) ?

Code: Select all

p.s=Space(18306000000)
Debug Len(p.s)

Yes, it is more than 18 billion bytes (billion in American English, it is 18 milliards in French, Dutch, German, Italian,...)...

[STARGÅTE]

Yes, it is more than 18 billion bytes (billion in American English, it is 18 milliards in French, Dutch, German, Italian,...)...

No, it is unicode, so 18 billion characters need 36 GB RAM, do you have more than 36 GB RAM?

[Demivec]

Code: Select all

p.s=Space(18306000000)
Debug Len(p.s)

Yes, it is more than 18 billion bytes (billion in American English, it is 18 milliards in French, Dutch, German, Italian,...)...

18306000000 = 18306000000 Unicode chars at 2 bytes a piece = 36612000000 bytes = 36612000 KB = 36120 MB = 36.12 GB.

Do you have 34.12 GB of free memory to create the string?

The string length is unlimited but your RAM is limited.

[charvista]

I have 64 Gb of RAM....

[NicTheQuick]

As you can see in his signature he has 64 GiB of RAM. This does not mean it is free but at least it is there.
However this only works if the program gets compiled with 64 bit compiler.

I tested it on my system where I've got 47 GiB of free RAM but the program crashed before it had allocated any memory.
It worked with half the characters (9153000000).

But then I tried this code:

Code: Select all

;       GGGMMMkkkBBB 
#size =   9500000000
p.s=Space(#size)
MessageRequester("Size", Str(Len(p.s)))

And for some reason the output was: 910065408

That means only 9,6% of the requested characters were there.

So what's the problem? Space() or Len()? I think it is Space(). Since Space() should allocate enough memory and also fill it with spaces the memory really should be consumed by the process. But I never saw the process acquiring so much memory. There was just a little bump in my memory graph.

When I try 10.000.000.000 characters the process just crashes.

[STARGÅTE]

Nevertheless, even if not enough space is possible to allocate for the string, Space() should not to generate an IMA, but just return an empty string.

[charvista]

@NicTheQuick
I tried your code, and I get the exact same result: 910065408
Demivec's and Stargåte's answer did me think about closing all running programs to free any using memory, but the problem is still there.

I tested more and found out that the maximum lies around 1'000'000'000, seen as Unicode, that's taking up only 2 GB of memory. That's very far away from the "unlimited" for a 64GB RAM under x64 processor.

Code: Select all

p.s=Space(1000000000)
Debug Len(p.s)
p.s=""
p.s=Space(1100000000)
Debug Len(p.s)
p.s=""

The first Debug will work, the second one crashes.

[mk-soft]

I think unlimit means a meaningful range.
So more than 64 kb and less than 1 mb with which the string management can also process well.

Anything in the gigabyte range should be processed directly with memory areas as ArrayOfChar.

Code: Select all

#GByte = 1024 * 1024 * 1024

Structure ArrayOfChar
  c.c[0]
EndStructure

size = 20 * #GByte

*mem.ArrayOfChar = AllocateMemory(size, #PB_Memory_NoClear)

If *mem
  Debug "Size = " + Str(MemorySize(*mem) / #GByte)
  FreeMemory(*mem)
Else
  Debug "Out of memory"
EndIf

[NicTheQuick]

I think unlimit means a meaningful range.
So more than 64 kb and less than 1 mb with which the string management can also process well.

Anything in the gigabyte range should be processed directly with memory areas as ArrayOfChar.

Code: Select all

<...>

Actually your code does nothing meaningful. I mean, yes, the memory is going to be allocated but if you don't write anything to it it will not be consumed and you won't see a spike in your memory usage.
Instead use FillMemory() and a MessageRequester() to pause the process before freeing the allocated memory again. Even without '#PB_Memory_NoClear' the memory usage will not raise because the kernel can mark memory as nulled even if it is not the case.
This works fine for me to see an actual increase in memory consumption:

Code: Select all

#GByte = 1024 * 1024 * 1024

Structure ArrayOfChar
	c.c[0]
EndStructure

size = 20 * #GByte

*mem.ArrayOfChar = AllocateMemory(size, #PB_Memory_NoClear)

If *mem
	FillMemory(*mem, size, $deadbeef, #PB_Long)
	MessageRequester("RAMeater", "Size = " + StrD(MemorySize(*mem) / #GByte, 3) + " GiB")
	FreeMemory(*mem)
Else
	MessageRequester("RAMeater", "Not enough food.")
EndIf

[charvista]

Yes, with *Memory, it is not a problem at all - below code works perfectly:

Code: Select all

#GByte = 1024 * 1024 * 1024
*mem=AllocateMemory(20 * #GByte)
Debug MemorySize(*mem)
PokeS(*mem+19999999950,"The FOOD is HERE",16,#PB_Unicode)
Delay(1000)
Debug PeekS(*mem+19999999950,16,#PB_Unicode)
FreeMemory(*mem)

But my problem was about the String variable. I believe it still has a 2GB barrier.
Because this works:

Code: Select all

p.s=Space(1000000000)
Debug Len(p.s)
q.s=Space(1000000000)
Debug Len(q.s)
r.s=Space(1000000000)
Debug Len(r.s)
s.s=Space(1000000000)
Debug Len(s.s)
t.s=Space(1000000000)
Debug Len(t.s)

And I do *not* clear the variables between them, so it is piled up in different variables.

[charvista]

And finally, even having a large *Memory is not of big help, because with WriteData(), the maximum buffer size is 2GB.
I got the PB message [ERROR] WriteData(): The maximum buffer size is 2GB.
PureBasic has obviously still barriers to 2GB...

[pjay]

The 2gb limit of WriteData() is documented - you can use multiple writes to overcome it.

[BarryG]

So what's the problem? Space() or Len()?

Maybe try with LSet() and see? @Charvista, try this. If it works, then I guess Space() is the problem?

Code: Select all

p.s=LSet("",18306000000," ")
Debug Len(p.s)

[charvista]

@pjay
Yes, that's true, it is documented. But I forgot it. Thank you for recalling me.

@BarryG
The problem is Space(). LSet() gives the same problem: [ERROR] Invalid memory access. (write error at address 39854080).

[pjay]

mk-soft was right when he said "I think unlimited means a meaningful range". Unlimited should not be taken literally & is perhaps not the best terminology to use, there are always limits.

Personally, I think anyone who's using a gigabytes worth of singular string data should be taking a good long look at themselves in the mirror