PB Strings, Can be serveral thousand times too slow

[pdwyer]

[This gets a little long but if you're interested in string performance especially in tight loops then I'd be happy if you'd take the time to read and join the discussion, putting our heads together could generate some great improvements]

In a couple of little project I've encountered some performance issues with strings but this one takes the cake. In making a design change to cope with a performance bottleneck I stumbled over something quite surprising.

I think many people have had a situation where they need to process a large amount of text, like a log file or something and need to split the file up on the #CRLF$. I hit a big problem because I was taking a shortcut to replace the CRLF with a single char then use the PB ParseString() function to extract the bit. For smallish files this is okay but as you go over a couple of thousand lines you notice a slowdown as it progresses and over 10's of thousands of lines you see it drop to a crawl.

Simple design problem on my part of course, ParseString() has to count up the CR's and when the line count gets to say 20,000 then it has to count that far just to process one more line.

Solution, a split() function to just go and get the bits once rather than recalc for each line. Can handle the two byte CRLF so you get rid of the replace too. Hardly rocket science

Looking through the forums, there are a few split functions, some use the parsestring() internally so they wouldn't solve my perf problem some don't but weren't quite what I wanted so I wrote my own. I based it on the quicksearch algorithm since it (unlike findstring() ) can keep going after it finds what it's looking for and maximise the use of it's badchar() table it builds at the beginning. It's also faster the larger the delimiter so crlf will search faster than just cr.

Strangely enough, using the split function was not faster unless there were over about 15,000 lines. I thought excessive ReDim'ing to increase the array was at fault but diming out a huge array before hand was the same and in the end I found it was the PB Mid() function on large strings that was slow. This seemed odd as the size parameters are passed so it should have the problem with the slow PB Len() function. Anyway, I replaced the PB mid function for this

Code: Select all

Procedure.s MidMem(*MainMem, StartPos.l, GetLen.l)  

    If *MainMem = 0 Or GetLen = 0
        ProcedureReturn ""
    EndIf
    
    *RetMem = AllocateMemory(GetLen)
    CopyMemory(*MainMem + StartPos -1, *RetMem, GetLen)

    ProcedureReturn PeekS(*RetMem,GetLen)

EndProcedure 

Now I'm sure PB's function does more checking, most probably an over run bounds check which then needs that evel len() calced which could be the cause, it would slow my function down enourmously to check this (unless we were using BStrs where the length was already available to us). This slightly more dangerous (I guess) function speeded up my split function x 10,000 times over the parsestring() method!! My Problem was solved

I decided to do a test just to see what the perf difference of the Mid() function was since the design change was part of the reason for the speedup being so great so I added this in:

Code: Select all

DisableDebugger

TestString.s = Space(20000000)
Output.s
Start1.l = ElapsedMilliseconds()

For i = 1 To 100000
    Output = "!" + Midmem(@TestString,i*100,10) + "!" 
Next

Start2.l = ElapsedMilliseconds()

For i = 1 To 100000
    Output = "!" + Mid(TestString,i*100,10) + "!"   
Next

Start3.l = ElapsedMilliseconds()
EnableDebugger
MessageRequester("Mid vs MidMem", "Mid() = " + Str(Start3-Start2) + #CRLF$ + "MidMem() = " + Str(Start2-Start1))

The result was:

---------------------------
Mid vs MidMem
---------------------------
Mid() = 902562
MidMem() = 31
---------------------------
OK
---------------------------

This is admittedly a very large string. Previous versions of PB with the 64k limit would have made people work around it in otherways so this would have been a much smaller issue, not sure how PB handled getting a 1mb text blob through the clipboard though

I'm not meaning this as a bash on PB. Their memory functions and pointer arithmatic really fly and make these workarounds possible but I think we need to spend some time flushing these thing out into the open so we can see the benefits. In a larger function where the use of something like MidMen() and it's inputs are controlled then the risk is quite small of memory overruns although accepting user input into it would be quite dangerous.

The Split function currently looks like this: (But I want to clean it up and merge the midmem() into it and get rid of the function call)

Code: Select all

Structure MemoryArray 
    Byte.c[0] 
    word.w[0]
EndStructure 

Procedure.l Split(StringArray.s(1), Text2Split.s, Delim.s) ;return count

    FindLen.l = Len(Delim)
    MainLen.l = Len(Text2Split)
    Dim StringArray.s(1000)
    StringCount.l = 0
    
    *MainByteArray.MemoryArray = @Text2Split  ;*MainMem
    *FindByteArray.MemoryArray = @Delim       ;*FindMem 

    PrevPos.l = 1

    ; Build BadChr Array
    Dim BadChar.l(255)
    
    ; set all alphabet to max shift pos (length of find string plus 1)
    For i = 0 To 255
        BadChar(i)  =  FindLen + 1
    Next
    
    ;Update chars that are in the find string to their position from the end.
    For i = 0 To findlen -1
        BadChar(*FindByteArray\byte[i]) = findlen - i    
    Next     

    MainArrayLoop.l = 1 
    EndSearchPos.l = MainLen - (FindLen -1)
    
    While MainArrayLoop <= EndSearchPos
    
        If CompareMemory(@Text2Split + MainArrayLoop, @Delim, FindLen) = 1
            FoundPos = MainArrayLoop + 1
            
            If StringCount % 1000 = 0  ; not really needed, doesn't have much of a speed increase. This used to do a lot in the old VB days         
                ReDim StringArray.s(StringCount + 1000)
            EndIf
                
            StringArray(StringCount) = MidMem(@Text2Split, Prevpos, Foundpos - PrevPos) ;Mid(Text2Split, Prevpos, Foundpos - PrevPos) ;"HEllo, this is some text" + #TAB$ + "  " + #TAB$ + "esdfsdf"
            StringCount = StringCount + 1
            PrevPos = foundpos + Findlen

        EndIf 
        ;Didn't find the string so shift as per the table.
        MainArrayLoop + BadChar(*MainByteArray.MemoryArray\byte[MainArrayLoop + FindLen])
    Wend
 
    ;catch end
    ReDim StringArray.s(StringCount)
    StringArray(StringCount) = Mid(Text2Split, Prevpos, MainLen - PrevPos +1)
    StringCount = StringCount + 1
    ReDim StringArray.s(StringCount)

    ProcedureReturn StringCount

EndProcedure

I feel that there must be more in this direction that can be investigated.

[tinman]

I can't remember if the length of a string gets stored anywhere, but if not the only way to perform bounds validation on the string would be to step through it and stop on the null terminator. That would be a source of significant slowdown.

But if you're dealing with unicode (which I'm guessing you aren't from the code you posted) then it's probably safer to step through the characters, since you can handle the code points which take up two UTF16 characters in the string.

Edit: UTF, not UCS.

[freak]

PB strings are 0-terminated, no size value is kept.
So obviously performing a high number of string operations on a large string will be slow.

For data of this size, i would use direct memory manipulation and no strings at all (if anything, then only to store the final results)
This way you are not only in control of the length issues, but also of the memory reallocations
that come with reassigning string values to different variables.

[pdwyer]

Yes, I'm aware of these things.

My point is that while functionally PB strings are fine, they can be quite a performance bottleneck over and above what you would expect from a string (unless PB was the only compiler you'd ever used I guess) and that there are workarounds.

But if the specific impacts of the performance degradation is not fully understood by the developer(s) then it's hard to know when and in what situations that the PB string engine should be dropped do solve performance problems.

This post was not intented as a message to PB to fix something or wishlist item or a complaint (so please don't take it like that). It is merely an invitation to peer discussion that we (the users) can build on to improve our apps.

A possible outcome could be something like a new OSL with a high performance but low checking string engine which people can use just for those situations where it's needed.

[tinman]

Yes, I'm aware of these things.

Sorry, I re-read your post and realised I'd missed parts. It was quite long though ;)

My point is that while functionally PB strings are fine, they can be quite a performance bottleneck over and above what you would expect from a string (unless PB was the only compiler you'd ever used I guess) and that there are workarounds.

But if the specific impacts of the performance degradation is not fully understood by the developer(s) then it's hard to know when and in what situations that the PB string engine should be dropped do solve performance problems.

Personnally, I'd always assume the worst about the strings performance, because the commands are very general purpose and can be called in any sequence.

freak has confirmed that strings do not store their length, so any string operation that uses character positions within the string will have to iterate the characters from the start of the string, checking for null to ensure it doesn't run off the end. Therefore string commands using character positions on big strings in big loops is always going to be a big performance hit.

A possible outcome could be something like a new OSL with a high performance but low checking string engine which people can use just for those situations where it's needed.

OSL? Open Source Library?

You could also collect the findings, and workarounds or new or improved algorithms and submit feature requests for each one. It would benefit everyone not just those that knew to go looking for an additional string library.

Some more restrictive or specific commands for operations at a level above the basic string functions would be useful and seem to be used/needed a lot, as you found with your mid and split functions. They'd be able to provide improved performance.

For example, if there were commands that had to be called in sequence (like, I dunno, StartStringField() and NextStringField() instead of StringField() with a variable index) then the commands could make some assumptions about the string. You could cache the length and the current position, and require that the string not be changed while it was being processed.

Another common sequence would be splitting by length, as you've done with Midmem(), but it's could use the same principle. Cache the length and current position, extract strings by length.

[Trond]

Yes, I'm aware of these things.

My point is that while functionally PB strings are fine, they can be quite a performance bottleneck over and above what you would expect from a string (unless PB was the only compiler you'd ever used I guess) and that there are workarounds.

PB strings are normal strings, so if you expect them to perform better than other 0-terminated strings, then you will of course be disappointed.

Edit: It's easy to create faster strings, but they would be incompatible with everyone else in this world.

[pdwyer]

Well, for 3 years I used VB then for 7 I used Powerbasic and both of these languages use BStrs which have the len built into the structure, their strings performance was a lot better. But if Bstrs are not built into the language then they are a pain in the butt to use so I don't think I'll touch them. Besides, pointers are faster than bstrs and PB does pointers well I think

I have some ideas for a string engine, something basic to test the concept. Be fast with memory but still easily usable with PB API calls etc. so I'm going to have a play with this.

Perhaps most people don't perceive it as much of an issue like me.

In the app I was building, a 2 min wait time to process tabbed text from the clipbloard into 50,000 SQLite rows turned into 11 seconds (10.5 of which were SQLite writing transactions).

I could have saved a lot of my time if I had some include file to process text more efficiently

[Trond]

I have some ideas for a string engine, something basic to test the concept. Be fast with memory but still easily usable with PB API calls etc. so I'm going to have a play with this.

The stupid thing about BStr is that it lacks reference counting.

I have already written a string engine that is faster than C++ (GCC) strings, but unfortunately it's not much use unless it's tightly intergrated into a language. It can't be used "by hand". I don't know if you are interested in it?

[pdwyer]

If it's in PB I'd be interested in taking a look. C++ spaghetti makes my head spin

Thanks

[Trond]

It's asm.

[JCV]

Im also interested.
Any chance in converting it as PB include?

[pdwyer]

It's asm.

I kinda suck at that.

I've been thinking about which way to go about this, the easy way or the hard way. The easy way would be to just create functions like the midmem function to do things a different way and return a normal string. It's a lot easier and it's a lot simpler to use in PB but some area's will be hard to get any performance increase from. (Right() for example is not going to get any faster if you end up having to look for the null terminator)

The hard way would be to made a string structure which, in order to handle functions would need to be a structure with a pointer to a structure which had a length and a pointer to the data. All the string functions would be able to work with the length and update it depending on what they did.

Other than the normal functions to do similar things to what PB does (left, right, mid, len, hex, find, replace etc) you would have to have a concaternator, and a function to dump a string to use normally and probable a pointer to the data to pass to APIs.

It's more work and more room for bugs but it would be better over all.

While I play with some of the functions I'm thinking about (procrastinating over ) which way to go

[Trond]

I'll post it anyways, because I'm so proud of it!
It works only with unicode characters. Each string is a structure with refcount, length and data (see the source).
String variables are long values that points to the string data in a string structure (they do not point to the start of the structure).

Code: Select all

; String manager interface
public rq_StrAlloc
public rq_StrCopy
public rq_StrFree
public rq_StrLen
public rq_StrConcat
public rq_StrConcatFreeA
public rq_StrConcatFreeB
public rq_StrComp
public rq_LongToStr

; String manipulation functions
public _LEN@4
public _MID@12
public _STR@4
public _STRU@4
_STR@4 = rq_LongToStr
_STRU@4 = rq_ULongToStr

; Required support functions
extrn rq_AllocateMemory
extrn rq_FreeMemory
extrn rq_CopyMemory
extrn "__imp__lstrcmpW@8" As _lstrcmp: dword

; The size of a rqString
; Four bytes: number of references
; Four next bytes: length of string in bytes
; Next bytes:  szString
rq_SIZEOF_EMPTYRQSTR equ 8
; Structure rqString
;   Refcount.l
;   Length.l (in bytes, not including 0)
;   String.c[0]
; EndStructure


_LEN@4:
pushd [esp+4]
call  rq_StrCopy
mov   ecx, [eax-4]
push  eax
mov   eax, ecx
sar   eax, 1      ; Unicode = divide by two
call  rq_StrFree
ret   4

; Mid(String [esp+4+4], StartPos [esp+4+8], Length [esp+4+12])
_MID@12:
pushd 0               ; [esp] New string
pushd [esp+4+4]
call  rq_StrCopy
mov   [esp+4+4], eax  ; Store copied string back
dec   dword [esp+4+8] ; String indexes starts at 1, so translate it to 0
sal   dword [esp+4+8], 1 ; Multiply by 2 for unicode
sal   dword [esp+4+12], 1
mov   ecx, [esp+4+8]  ; if StartPos < 0 -> StartPos = 0
cmp   ecx, 0
jnl   l_FixStartDone
mov   dword [esp+4+8], 0
l_FixStartDone:
mov   eax, [eax-4]    ; Length of original string
sub   eax, [esp+4+8]  ; OrigLen - StartPos
cmp   eax, [esp+4+12]
jnl   l_NewLen_Done   ; Original string not long enough for length
mov   [esp+4+12], eax ; Cut the Length variable
l_NewLen_Done:
pushd [esp+4+12]      ; Allocate new string
call  rq_StrAlloc
mov   [esp], eax
mov   ecx, [esp+4+12]
mov   [eax-4], ecx    ; Set length of new string
mov   eax, [esp+4+4]  ; CopyMemory(String+StartPos, NewString, Length)
add   eax, [esp+4+8]
mov   ecx, [esp]
pushd [esp+4+12]
push  ecx
push  eax
call  rq_CopyMemory
pushd [esp+4+4]
call  rq_StrFree
mov   eax, [esp]
add   esp, 4
ret   12

; --------------------------------------------
; StringManager: rq_StrAlloc
;       Create a rqString
; In:  BufferByteLength AS INT
; Out: StringPtr AS rqString
rq_StrAlloc:
add   dword [esp+4], rq_SIZEOF_EMPTYRQSTR + 6 ; Two bytes for null terminator, 4 for padding
push  dword [esp+4]
call  rq_AllocateMemory
mov   dword [eax], 1              ; Reference count is 1
add   eax, rq_SIZEOF_EMPTYRQSTR   ; Return address of buffer
ret   4


; --------------------------------------------
; StringManager: rq_StrFree
;       "Client-is-not-reference-aware" rqString free function
; In: Pointer AS rqString [esp+4]
; Out: EAX (preserved)
rq_StrFree:
push  eax
mov   eax, [esp+8]
cmp   eax, 0              ;; If the string is a null string
jz    rq_StrFree_End      ;; we don't have to do anything
sub   eax, 8
cmp   dword [eax], 1
je    rq_StrFree_FreeMemory   ;; if refcount = 1 -> free the memory
cmp   dword [eax], -1
je    rq_StrFree_End          ;; if refcount = -1 -> static string, don't free
sub   dword [eax], 1          ;; else, decrease the refcount
jmp   rq_StrFree_End
rq_StrFree_FreeMemory:
push  eax     ;; Push base of memory
call  rq_FreeMemory
rq_StrFree_End:
pop   eax
ret   4

; --------------------------------------------
; StringManager: rq_StrCopy
;       "Client-is-not-reference-aware" rqString copy function
; In: Pointer AS rqString [esp+4]
; Out: String as rqString
rq_StrCopy:
mov   eax, [esp+4]
cmp   eax, 0              ;; If the string is a null string
jz    rq_StrCopy_End      ;;      we don't have to do anything
cmp   dword [eax-8], -1   ;; If the string is a literal string
je    rq_StrCopy_Copy     ;;      we need a physical copy
add   dword [eax-8], 1    ;; Else, just increase refcount
rq_StrCopy_End:
ret   4
rq_StrCopy_Copy:
push  eax
pushd [eax-4]             ;; Byte length
call  rq_StrAlloc
pop   ecx                 ;; Old string in ecx, new in eax
mov   edx, [ecx-4]
mov   [eax-4], edx        ;; Setup byte length for the new string
push  eax
pushd [ecx-4]             ;; Byte length
push  eax
push  ecx
call  rq_CopyMemory
pop   eax
jmp   rq_StrCopy_End

; --------------------------------------------
; StringManager: rq_StrConcat
;       Return a new string (a + b)
; In: A, B AS rqString [esp+12+4], [esp+12+8]
; Out: NewString AS rqString
rq_StrConcat:
;; Check for null strings
cmp   dword [esp+4], 0
je    rq_StrConcat_ReturnB
cmp   dword [esp+8], 0
je    rq_StrConcat_ReturnA
jmp   rq_StrConcat_Main
rq_StrConcat_ReturnB:
pushd [esp+8]
call  rq_StrCopy
ret 8
rq_StrConcat_ReturnA:
pushd [esp+4]
call  rq_StrCopy
ret 8
rq_StrConcat_Main:
;; Main rq_StrConcat:
push eax  ; [esp+8]   = length of A
push eax  ; [esp+4]   = length of B
push eax  ; [esp+0]   = new string
mov eax, [esp+12+4]
mov eax, [eax-4]  ; Length of A
mov [esp+8], eax
mov ecx, [esp+12+8]
mov ecx, [ecx-4]  ; Length of B
mov [esp+4], ecx
add eax, ecx      ; Length of A + B
push eax          ; Push for later
push eax
call rq_StrAlloc  ; Alloc new string
mov [esp+4], eax  ; [esp+0+stored eax]
popd [eax-4]      ; Setup length of new string (A+B)
; Copy A
mov   ecx, [esp+12+4]
pushd [esp+8]
push  eax
push  ecx
call  rq_CopyMemory
; Copy B
mov   ecx, [esp+12+8]
mov   eax, [esp+0]
add   eax, [esp+8]
pushd [esp+4]
push  eax
push  ecx
call  rq_CopyMemory
mov   eax, [esp+0]
add   esp, 12
ret   8

; --------------------------------------------
; StringManager: rq_StrConcatFreeA
;       Return a new string (a + b) and free a
;       Todo: Optimize by adding b directly onto a
; In: A, B AS rqString [esp+4], [esp+8]
; Out: NewString AS rqString
rq_StrConcatFreeA:
pushd [esp+8]     ; Push B
pushd [esp+8]     ; Push A (offsetted)
call  rq_StrConcat
pushd [esp+4]
call  rq_StrFree  ; Free A
ret   8

; --------------------------------------------
; StringManager: rq_StrConcatFreeB
;       Return a new string (a + b) and free b
; In: A, B AS rqString [esp+4], [esp+8]
; Out: NewString AS rqString
rq_StrConcatFreeB:
pushd [esp+8]     ; Push B
pushd [esp+8]     ; Push A (offsetted)
call  rq_StrConcat
pushd [esp+8]
call  rq_StrFree  ; Free B
ret   8

; --------------------------------------------
; StringManager: rq_StrComp
;       Compare two strings, return 0 for equal, else negative or positive
; In: A, B AS rqString [esp+4], [esp+8]
; Out: Bool AS INT
rq_StrComp:
;; Check for empty strings. If either string is empty, go to special null
;;  string check
mov   eax, [esp+4]
cmp   eax, 0
je    rq_NullStringCheckB
cmp   dword [eax-4], 0          ;; Length is 0?
je    rq_NullStringCheckB
mov   eax, [esp+8]
cmp   eax, 0
je    rq_NullStringCheckA
cmp   dword [eax-4], 0          ;; Length is 0?
je    rq_NullStringCheckA
;; Neither string is an empty or null string, do a real comparison:
pushd [esp+8]
pushd [esp+8]
call  [_lstrcmp]
rq_StrComp_End:
ret   8   ;; EXIT 2
rq_StrComp_Return0:
mov   eax, 0
ret   8   ;; EXIT 1

rq_NullStringCheckA:  ;; String B is null or empty, is A as well?
mov   eax, [esp+4]
cmp   eax, 0          ;; Yes eax (A) is 0, return 0 for equal
je    rq_StrComp_End
cmp   dword [eax-4], 0    ;; Is the length of A 0?
je    rq_StrComp_Return0
mov   eax, 1          ;; A > B, positive return value
jmp   rq_StrComp_End

rq_NullStringCheckB:  ;; String A is null or empty, is B as well?
mov   eax, [esp+8]
cmp   eax, 0          ;; Yes eax (B) is 0, return 0 for equal
je    rq_StrComp_End
cmp   dword [eax-4], 0    ;; Is the length of B 0?
je    rq_StrComp_Return0
mov   eax, -1          ;; A < B, negative return value
jmp   rq_StrComp_End

; --------------------------------------------
; StringManager: rq_StrLen
;       Return the length in characters of a string
;       NOT for use with loaded strings, it does not free it
;       Returns the length in BYTES, not characters
;       Use _LEN@4 for loaded strings
; In: Pointer AS rqString [esp+4]
; Out: Length AS INT
rq_StrLen:
mov   eax, [esp+4]
mov   eax, [eax-4]
ret 4

; --------------------------------------------
; StringManager: rq_ULongToWideBuffer_Customcall  : UNSIGNED
; ; eax: scratch
; ; edx: div (scratch)
; ; ecx: num    IN
; ; edi: buffer IN
; Out: Bytes written
rq_ULongToWideBuffer_Customcall:
push esi
XOr  esi, esi
loopstart:
  mov	eax, -858993459   ; edx, div = num / 10
  mul	ecx
  shr	edx, 3

  lea  eax, [edx+edx*8]  ; rem, ecx = num - div*10
  add  eax, edx
  sub  ecx, eax

  lea  eax, [ecx+'0']    ; S.s + Chr(Rem+'0')
  mov  word [edi+esi], ax
  add  esi, 2

  cmp  edx, 0
  je   loopend
  mov  ecx, edx          ; num = div
jmp  loopstart
loopend:
XOr ecx, ecx

push esi
turnstart:               ; ecx = 0
  add  esi, -2
  mov  ax, word [edi+esi]
  mov  dx, word [edi+ecx]
  mov  word [edi+ecx], ax
  mov  word [edi+esi], dx
  add  ecx, 2
  cmp  esi, ecx
jg turnstart
pop eax
pop esi
ret

; --------------------------------------------
; StringManager: rq_LongToWideBuffer_Customcall : SIGNED
; ; eax: scratch
; ; edx: div (scratch)
; ; ecx: num    IN
; ; edi: buffer IN
; Out: Bytes written
rq_LongToWideBuffer_Customcall:
cmp   ecx, 0
mov   dword [v_ADDEDMINUS], 0
jge   nominus
mov   word [edi], '-'
neg   ecx
add   edi, 2
mov   dword [v_ADDEDMINUS], 1
nominus:
call  rq_ULongToWideBuffer_Customcall
cmp   dword [v_ADDEDMINUS], 1
je    AddMinus
ret
AddMinus:
adc   eax, 2
ret
new AddedMinus As LONG

; --------------------------------------------
; StringManager: rq_LongToStr
; In: Int AS LONG [esp+4]
; Out: Int AS STRING eax
rq_LongToStr:
push 64    ;; Max length of new string, warning: longer than Len(Str(Maxint)) is
            ; required for rq_SingleToStr, which depends on this
            ; 11*2 -> 21 bytes in unicode is Len(Str(MAXINT))
call rq_StrAlloc
push edi
mov  edi, eax
mov  ecx, [esp+4+4]
push eax
call rq_LongToWideBuffer_Customcall
pop  edi
mov  [edi-4], eax      ;; Set the length of the string
mov  eax, edi
pop  edi
ret  4

; --------------------------------------------
; StringManager: rq_ULongToStr
; In: Int AS LONG [esp+4]
; Out: Int AS STRING eax
rq_ULongToStr:
push 22
call rq_StrAlloc
push edi
mov  edi, eax
mov  ecx, [esp+4+4]
push eax
call rq_ULongToWideBuffer_Customcall
pop  edi
mov  [edi-4], eax      ;; Set the length of the string
mov  eax, edi
pop  edi
ret  4

To use the string manger properly you need to understand three "kinds" of strings:
- Normal strings ("String")
- Loaded strings
- Literal strings

All of the strings have the same physical format, but the literal strings have a refcount of -1, and will never be freed. Apart from that it's handled as a normal string.

Normal strings are strings that are assigned to a string variable.

Loaded strings are strings that are NOT assigned to a variable. They come from string computation. If you add two normal strings, the result is a loaded string. You then need to assign that string to a string variable and it becomes a normal string.
There is no physical difference, only a difference in handling. You must be sure to do something (either free or assign) your loaded string or you will get a memory leak.

It sounds complicated, but it's actually much easier to understand than PB's string system.

Let's look at an example:

Code: Select all

c.s = a.s + b.s
; becomes this:
        push   dword [v_B]
        push   dword [v_A]
        call   rq_StrConcat ; Return the loaded string A+B

        push   dword [v_C]
        call   rq_StrFree ; Free the old string in C

        mov    dword [v_C], eax ; Assign the loaded string to C
; (rq_StrFree preserves eax)

Much easier to understand than PB's code:

Code: Select all

  MOV    edx,dword [v_a]
  PUSH   dword [_PB_StringBasePosition]
  CALL  _SYS_CopyString@0
  MOV    edx,dword [v_b]
  CALL  _SYS_CopyString@0
  LEA    ecx,[v_c]
  POP    edx
  CALL   SYS_AllocateString

If you want to add three strings (c = a + b + c) then a + b is calculated first. When you add (a+b) to c you don't need (a+b) afterwards. There is a special function for this: rq_StringConcatFreeA, which works like rq_StringConcat except that the first argument is freed after the concatenation.

Code: Select all

	push   dword [v_B]
	push   dword [v_A]
	call   rq_StrConcat ; return a + b

	push   dword [v_C]
	push   eax ; this is what was just returned
	call   rq_StrConcatFreeA ; return (a+b)+c and free (a+b)

	push   dword [v_C]
	call   rq_StrFree ; Free the old value in C

	mov    dword [v_C], eax ; Assign the result of StrConcatFreeA to C

[pdwyer]

cheers, I'm gonna spend some time with this, I won't understand the meat code but I'm interested in the design

[Trond]

You can have a look at the generated code from my compiler to see how it's handled with functions and more complicated expressions: http://home.no.net/tsg1zzn/c21.zip