Page 1 of 1

Fastest strings concatenation

Posted: Fri Sep 01, 2017 5:59 pm
by Lunasole
Oh I know it was many times on forum, just I've finally tired to code that bicycle everytime and made generic macro/procedures for it ^^
The following is CLOSER TO perfect :3 (comparing with previous dirty version).
You can easily change that code to be thread-safe, or pack it into module, and so on with what is needed

Code: Select all

EnableExplicit

;{ StrFastConcat } 
	
	; v 1.0.0.5
	;   2017		(c) Luna Sole
	Structure STRFASTOBJ
		*Data
	EndStructure
	
	; Starts new concatenation. Lot of them may exists and be threaded
	; RETURN:		strfast object on success
	Procedure StrFastNew()
		Protected *StrFast.STRFASTOBJ = AllocateStructure(STRFASTOBJ)
		If *StrFast
			*StrFast\Data = AllocateMemory(4 + 10240*SizeOf(Character))
			If *StrFast\Data
				PokeL(*StrFast\Data, 0)      ; bytes 1-4 storing current len (chars)
				ProcedureReturn *StrFast
			Else
				FreeStructure(*StrFast)
			EndIf
		EndIf
	EndProcedure
	
	; Concatenate strings quite optimized way
	; *String		pointer to a string
	; Len			*String len (chars). If set, len is not calculated here, which brings some more speed in some cases
	; RETURN:		none
	Procedure StrFastAddEx (*StrFast.STRFASTOBJ, *String, Len = 0)
		Protected NewLenB, *FastLen.Long
		If *StrFast
			If Len <= 0
				Len = MemoryStringLength(*String)
			EndIf
			If Len
				*FastLen = *StrFast\Data
				NewLenB = (*FastLen\l + Len)*SizeOf(Character)
				If NewLenB > (MemorySize(*StrFast\Data)-4)
					; every time buffer size is set to required size*2. that GREATLY raises both memory overhead and speed, comparing to use of small/medium fixed step
					; idea stolen from @Karig variant ^^
					*StrFast\Data = ReAllocateMemory(*StrFast\Data, 4 + NewLenB*1.28, #PB_Memory_NoClear)
					*FastLen = *StrFast\Data
				EndIf
				CopyMemory(*String, *StrFast\Data+4 + *FastLen\l*SizeOf(Character), Len*SizeOf(Character))
				*FastLen\l + Len
			EndIf
		EndIf
	EndProcedure
	
	; Wrapper for a cases where pointer is not applicable.
	; It is much slower than StrFastAddEx(), but only comparing to it. Itself is still very fast.
	; String$		a string to add
	; Len			String$ len (chars). If set, len is not calculated here, which brings some more speed in some cases
	; RETURN:		none
	Procedure StrFastAdd(*StrFast.STRFASTOBJ, String$, Len = 0)
		StrFastAddEx(*StrFast, @String$, Len)
	EndProcedure
	
	; Returns concatenated string
	; RETURN:		string
	Procedure$ StrFastGet(*StrFast.STRFASTOBJ)
		If *StrFast
			ProcedureReturn PeekS(*StrFast\Data+4, PeekL(*StrFast\Data))
		EndIf
	EndProcedure
	
	; Release specified fast string
	; RETURN:		none
	Procedure StrFastFree(*StrFast.STRFASTOBJ)
		If *StrFast
			FreeMemory(*StrFast\Data)
			FreeStructure(*StrFast)
		EndIf
	EndProcedure
	
;}


;;; Example

; Allocate this fast stuff
Define FS = StrFastNew()

; 1) Typical usage
StrFastAdd(FS, "test ")
; 2) Sometime this is faster, when is reasonable
StrFastAdd(FS, "this ", 5)
; 3) Using pointer + fixed len, fastest option here
StrFastAddEx(FS, @"ssst++", 4)

; Get Result & Release
Debug StrFastGet(FS)
StrFastFree(FS)

Re: Faster strings concatenation

Posted: Fri Sep 01, 2017 8:56 pm
by Kwai chang caine
Can be usefull
Thanks for sharing 8)

Re: Faster strings concatenation

Posted: Mon Oct 16, 2017 7:01 pm
by Lunasole
Posted updated version based on allocator.
It's universal and much faster comparing to previous crude stuff.

You're welcome with your enhancements to this 4-wheel bicycle ^^

Re: Faster strings concatenation

Posted: Mon Oct 16, 2017 8:24 pm
by infratec
Hi,

only one question:

why not

Code: Select all

Structure STRFASTOBJ
  len.l
  Step.l
  *Data
EndStructure
This would save some math :wink:

Bernd

Re: Faster strings concatenation

Posted: Mon Oct 16, 2017 9:16 pm
by Lunasole
infratec wrote:Hi,

only one question:
Hi. Mostly just because I don't like to add custom structures when can not add ^^
Generally even that STRFASTOBJ is rudimentary and left only "as decoration", can be nicely replaced with *Pointer.Integer/Integer structure.

Also anyway that math plays no role, should be 2-15ms speed up per million calls.

Re: Faster strings concatenation

Posted: Fri Oct 20, 2017 2:03 am
by Lunasole
@infratec, that pointer math should be avoided not for performance, but because of chances to shoot own legs, etc ^^

Just updated code to 1.0.0.3 with following patch to fix trivial-but-not too obvious bug (as allocated memory size grows, pointers defined previously in procedure may become invalid after reallocation):

Code: Select all

If NewLenB+8 > (MemorySize(*StrFast\Data))
	*StrFast\Data = ReAllocateMemory(*StrFast\Data, 8+NewLenB + *FastStepB\l)
	*FastLen = *StrFast\Data : *FastStepB = *StrFast\Data+4; ++
EndIf

Re: Faster strings concatenation

Posted: Fri Oct 20, 2017 4:16 pm
by Lunasole
Updated it once more. Now it's probably fastest variant on forum, except maybe something coded in ASM if any exists.

Though using large fixed step would be anyway faster (or equal), this variant is more usable for most cases, than setting astronomical step values.

Re: Fastest strings concatenation

Posted: Sun Oct 22, 2017 1:29 am
by Lunasole
And again changed a bit:
- added #PB_Memory_NoClear flag (which may bring some more speed, and maybe not)
- slighty balanced realloc coefficient: 1.28 acts better than 2.0, no such a crazy overhead at really large strings, while performance is not much affected

I also though to return option with fixed alloc step, but it really doesn't looks useful comparing with "auto" *1.28.
Generally now it should be realy done and stable, already used it in few programs.

Re: Fastest strings concatenation

Posted: Mon Oct 23, 2017 1:03 pm
by Thunder93
Looking very slick Lunasole, thanks for sharing :!:

Re: Fastest strings concatenation

Posted: Mon Oct 23, 2017 6:39 pm
by Lunasole
Thunder93 wrote:Looking very slick Lunasole, thanks for sharing :!:
np, but why "slick"? ^^

Re: Fastest strings concatenation

Posted: Mon Oct 23, 2017 11:46 pm
by Thunder93
Just thought it was appropriate here, best I felt describes It.

"(of an action or thing) done or operating in an impressively smooth, efficient, and apparently effortless way." :wink:

Re: Fastest strings concatenation

Posted: Tue Oct 24, 2017 5:09 am
by Lunasole
Thunder93 wrote:Just thought it was appropriate here, best I felt describes It.

"(of an action or thing) done or operating in an impressively smooth, efficient, and apparently effortless way." :wink:
Incompleteness of my english ^^ Had different association to that word and surprised from it in such context

Re: Fastest strings concatenation

Posted: Thu Feb 11, 2021 12:05 am
by Lunasole
Dam'n lol, I can't read some my old forum comments without facepalm.
My english was really mad (even though it is not perfect for today). Especially when I was replying in english while having some extremely good mood.....