> 64k Strings Hack - Does it work or not?

[Karbon]

So what is the final word - does this really enable > 64000 byte strings *AND* string handling (all the String library functions?)?

If it does and it also doesn't affect performance (as has also been said) then why not just set it to a couple of megs by default?

Code :

Code: Select all

Procedure SetStringManipulationBufferSize(size); - Set the String-Buffer-Size
  
  PBStringBase.l = 0 
  PBMemoryBase.l = 0 
  
  !MOV eax, dword [PB_StringBase] 
  !MOV [esp+4],eax 
  !MOV eax, dword [PB_MemoryBase] 
  !MOV [esp+8],eax 
  
  HeapReAlloc_(PBMemoryBase, #GMEM_ZEROINIT, PBStringBase, size) 
  
  !MOV dword [_PB_StringBase],eax 
  
EndProcedure 

Code taken from System.pbi that comes with jaPBE..

[fweil]

...,

This works but be careful.

If you change the Heap allocation, this changes the string length for all the following string declares, except if you split it back to the 'regular' value in the meanwhile.

And, I don't know ATM about 3.91, but it does not make string safer IMHO.

Rgrds

[Karbon]

changes the string length for all the following string declares,

I don't know what that means. What affect could this have on performance, memory usage etc?

[fweil]

Consequences are only memory oriented !

[Karbon]

Sure, but only for string space I *use*, right?

If I use the above code to set a 10 meg size it doesn't automatically make every string I declare 10 megs in size regardless of how much data is actually reference by the given string - right?

What other kinds of memory operation might be affected? Slow downs and resource usage would occur if I was actually passing all that data around - but just having it possible shouldn't affect much, if anything - right?

To summarize : Why isn't this method used by default to increase the buffer size for all strings by default. Why is the limit 64k? Is that some arbitrary number or is there a significance to 64k? There has to be some detrimental reason why Fred doesn't shut all us whining people up with a quick and easy increase of the buffer size to something crazy (like 100 megs or something)..

Thanks for the reply!

[fweil]

Only for the used space, I agree.

Why the method is not applied up to now, I don't know, but I imagine that the compiler is not an easy software to write.

...

[Karbon]

Oh no, I'm not saying that it's easy.. But this hack is waaay too simple for it to really work... If increasing all the possible string size was this easy in reality then I'd think Fred would go ahead and do it.. I guess this question is more directed to Fred than anyone.. So.....

Fred: Why isn't this the default behavior for PB? What does this do that makes it a bad idea that it happen by default? I know there is a reason and I'm just wondering what that reason is so I can decide if it's worth the risk!

Thanks!

[LarsG]

I think PB actually allocates the amount of memory as the size of you strings.. (ie. normally 64k.. more/less if you change it..)
(not 100% sure though) :roll:

[fweil]

Karbon,

As far as I remember, I wrote compilers and parsers on my own, string allocation makes some basic issues to bypass if you want your product to be performant either from the speed of execution, from the consumed memory and from the user's (coder's) points of view.

One way is simple to execute when compiling, to build bytes arrays, with the total requested amount of characters, but then when loading the executable, this will consume all space, even if you don't use it.

Then most of compilers attempt to make / use dynamic allocation, but then it is really difficult to negociate with the operating system, and more if you expect to have a multi-platform compiler.

Using Unix systems make possible to have less granularity issues than using MS Windows platform for example (even though it depends on what Unix based platform you are talking about). MS Windows memory granularity is 64K based. I guess this is the reason why a 64K string size is the 'default' for Purebasic. Because if you reserve a page for doing anything you want, it will consume 64K. By using a page allocation to store any characters in it consumes 64K even if the system does not tell you that 64K were consumed.

When using long, word, byte allocation, you reserve a memory address that is callable by any memory to register instruction. By defining a string object, you allocate a list of contiguous or non contiguous bytes depending on systems and methods. Up to the compiler to manage at the machine level what the programmer's intention was !

When you allocate a byte array of 64KB, you know that it should be safe using MS Windows, because the full page is available.

Then questions come up to the forum, because what should be is not exactly what is ...

[Karbon]

Interesting, that makes a lot more sense!

[blueznl]

well, it's all interesting, but as purebasic is a win32 application, i don't think the 64 kbyte issue applies anymore... in fact, i even think i once saw fred mention that 64000 was 'a nice number for a string limit' or something amongst those lines... (fred gates, anyone? )

[Karbon]

Damnit, Fred, reply to this right now!

[blueznl]

hey karbon, did you do a *search* on the forum?

:roll:

[Fred]

- yes, the 64000 bytes limit is arbitriary only, using the above routine just raise the limit..

[Karbon]

Great news Fred, thanks!!

Why not set the limit to at least a few megs - that would shut up %99.9 of the people that complain about this limit (me)!