PureBasic Forums - English

http://www.purebasic.com
https://www.purebasic.fr/english/

String manipulation with ${escape sequences}

https://www.purebasic.fr/english/viewtopic.php?t=84528

Page 1 of 1

String manipulation with ${escape sequences}

Posted: Thu Jun 13, 2024 11:24 am
by benubi
Hello!

I found a "frightening" bug on Linux. It has the strongest effect on the ARM-64 version (PB6.00 still because I need to change OS), but it also happens on Debian with PB6.11 (64 bit), when wildcard strings are being parsed recursively.

I use strings that I parse and contain wildcards in the form ${variable-or-function a b c} directly inspired from what I saw in Linux, PHP etc what have you. But when I try to parse those strings, which would remove all those $ in the output, the string gets "pre-parsed" and escaped in some way I can't figure out (mostly on arm-64 linux, sometimes the other linux, but NEVER on windows x86/x64).

Now here is where it gets creepy and ghosty: this happens when the console gets opened, and some parser procedures use the output (print) and get more corrupted/ghostly-escaped strings that those parts where the string manipulation doesn't "touch" Print calls. As if something would "forsee" that ${variables} could/would be used, or maybe there's a compiler switch and "ghosty things" happen there. Maybe only "because" of those escape/wildcard sequences.

Here you can compare the results, there are no visible problems on the Debian version but they will happen eventually.
https://ibb.co/1MCV5m2

I can send via network strings parsed on the raspberry to the debian machine, and the debian machine will parse & display them correctly, because on the raspberry side those strings are only assembled but not parsed+printed... it bypasses the Console-Procedures code parts which would print the result or change colors etc.

In the output strings no $ character should be seen since I use it only for formatting/wildcarding in the strings for now, to print it just use two $$ like in other standards.

Re: [PB6.xx] String manipulation bug with ${escape sequences}

Posted: Thu Jun 13, 2024 5:24 pm
by Fred
I don't understand your report, please post some code. To me it seems like a terminal issue when using printn() ?

Re: [PB6.xx] String manipulation bug with ${escape sequences}

Posted: Fri Jun 14, 2024 11:52 am
by benubi
Well I only have a bit of spaghetti code (in part)

It happens when $ dollar signs appear in a string. It seems cut out the first appearances of $ with a few chars after that $, and then escape the following $ in the string like that: $ -> $$. But that is only my impression.

The Parsing happens "recursively" with ParseWildcards() / ParseWildcards_Back() and the argument exploder functions ExplodeArgString() and ExplodArgstring2().

Both those function pairs use the lookup functions to fill the wildcards/parameter arrays,
and the lookup functions use those functions too.

The "line input" procedures I wrote are not working 100% on linux, backspace and cursor keys aren't, and on Windows there might be glitches especially in the password input field. That's not the focus but you can simply hit return 3 times and you are through the test program.

On windows (all backends/x86/x64): no errors - the title/head is well drawn with - + | characters around the username

Linux x64: colors work, inputs work (tho yet incomplete), but the head/title is corrupted

Linux arm64: does not work at all, all strings are "corrupted"

I thought of "DangerousSmiley$" to be the problem but it isn't; it shows in all debug windows and even the Linux terminal; only windows (10) terminal prints an "undefined" symbol.

Code: Select all

; =====================================================
; = UnescapeString2.pb
; =====================================================
; Author: benubi
; Filename: UnescapeString2.pb
; Status: Secret
;
; Declare$   _UnescapeString(String$)
; Declare$   _UnescapeString2(String$, LookUp.UE_LookupProcedure, *udata)
; Declare.i   _UnescapeString_Back(String$, *udataWriter, *udataLookup, Writer.UE_WriterProcedure, Lookup.UE_LookupProcedure)
;
; Declare   ExplodeArgString(String$, Array Result.s(1))
;           + a few little helpers
;
;
;   \a: alarm           Chr(7)
;   \b: backspace       Chr(8)
;   \f: formfeed        Chr(12)
;   \n: newline         Chr(10)
;   \r: carriage return Chr(13)
;   \t: horizontal tab  Chr(9)
;   \v: vertical tab    Chr(11)
;   \": double quote    Chr(34)
;   \\: backslash       Chr(92)
;
;
;++ extended part ++
;
;      \e: escape            Chr(27)
;    \xnn: ascii hex         Chr(0-255)
;  \unnnn: unicode hex       Chr(0-65535)
;    \nnn: octal digits[0-7] Chr(0-255)
;      \0: octal null-byte   Chr(0)
;
;
;  Lookup: ${variable-name}
;          ${var "name" 'value' some-thing}
;          ${x}
;          $[with brackets]
;          $(with parenthesis)
;          ${pretty}
;          $short   
;
;  Lookup2: with AND without parenthesis:
;          $varname_until_unfitting_char.
;          The lookup strings remain escaped at lookup function call time
;
; to do/want:

;
;
; "maybe" (low probability of implementation)
; c++ 23 conventions
;
;  \0{nnnnn,...}
;  \x{nnnnn,...}
;  \u{nnnnnn...}
;
;
EnableExplicit

; Like EplodeArgString() except that the results are parsed with a lookup procedure

; Prototype for lookup procedures
Prototype$ UE_LookupProcedure(*udata, key$, parenthesis, isWriter)

; Prototype for writer (output) procedures
Prototype UE_WriterProcedure(*udata, *buff, bytes) ; returns 0 to continue, other to abort

; Prototype for reader (input) procedures
Prototype UE_ReaderProcedure(*udata, *buff, bytes) ; returns 0 to continue, other to abort


Structure LookupStruct
  *VT ; = @LookupStruct\LookUp
  *LookupCookie
  *WriterCookie
  Lookup.UE_LookupProcedure
  Writer.UE_WriterProcedure
EndStructure

; Extended UnescapeString procedure
Procedure$ _UnescapeString(String$)
  Protected resultLength
  Protected *C.Character = @String$
  Protected octalvalue, i
  If String$=#Null$
    ProcedureReturn ""
  EndIf 
  ;   \a: alarm           Chr(7)
  ;   \b: backspace       Chr(8)
  ;   \f: formfeed        Chr(12)
  ;   \n: newline         Chr(10)
  ;   \r: carriage return Chr(13)
  ;   \t: horizontal tab  Chr(9)
  ;   \v: vertical tab    Chr(11)
  ;   \": double quote    Chr(34)
  ;   \\: backslash       Chr(92)
  ;++
  ;      \e: escape          Chr(27)
  ;      \0: null byte       Chr(0)
  ;    \nnn: octal           1-3 digits [0-7]
  ;    \x##: ascii hex       Chr(0-255)
  ;  \u####: unicode hex     Chr(0-65535)
  ;
  While *C\c
    If *C\c = '\'
      *C + SizeOf(Character)
      Select *C\c
        Case '0' To '9'
          i = 0
          While i < 3 And *C\c >= '0' And *C\c <= '9'
            i + 1
            *C + SizeOf(character)
          Wend
          resultLength + 1
        Case 'a', 'b', 'f', 'n', 'r', 't', 'v', 32, 39, 92, 'e'
          resultLength + 1
        Case 'x', 'X' ; ascii character (hex)
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          resultLength + 1
        Case 'u', 'U' ; unicode character
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          resultLength + 1
        Case 0
          Break
        Default
          resultLength + 1
      EndSelect
      
    Else
      resultLength + 1
    EndIf
    *C + SizeOf(Character)
  Wend
  
  Protected result$ = Space(resultLength)
  Protected *Z.Character = @result$
  Protected num.c
  *C = @String$
  
  While *C\c
    If *C\c = '\'
      *C + SizeOf(Character)
      Select *C\c
        Case '0' To '9' ; octal
          octalvalue = 0
          While *C\c >= '0' And *C\c <= '9' And i < 3
            octalvalue * 8
            octalvalue + *c\c - '0'
            *C + SizeOf(character)
            i + 1
          Wend
          *Z\c = octalvalue
        Case 'a'
          *Z\c = 7
        Case 'b'
          *Z\c = 8
        Case 'f'
          *Z\c = 12
        Case 'n'
          *Z\c = 10
        Case 'r'
          *Z\c = 13
        Case 't'
          *Z\c = 9
        Case 'v'
          *Z\c = 11
        Case 32, 39, 92
          *Z\c = *c\c
        Case 'e'
          *Z\c = 27
        Case 'x', 'X' ; ascii character (hex)
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          Select *c\c
            Case '0' To '9'
              num = *c\c - '0'
            Case 'a' To 'f'
              num = *c\c - 'a' + 10
            Case 'A' To 'F'
              num = *c\c - 'A' + 10
          EndSelect
          num << 4
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          Select *c\c
            Case '0' To '9'
              num = num | (*c\c - '0')
            Case 'a' To 'f'
              num = num | (*c\c - 'a' + 10)
            Case 'A' To 'F'
              num = num | (*c\c - 'A' + 10)
          EndSelect
          *Z\c = num
          
        Case 'u', 'U' ; unicode character
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          Select *c\c
            Case '0' To '9'
              num = *c\c - '0'
            Case 'a' To 'f'
              num = *c\c - 'a' + 10
            Case 'A' To 'F'
              num = *c\c - 'A' + 10
          EndSelect
          num << 4
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          Select *c\c
            Case '0' To '9'
              num = num | (*c\c - '0')
            Case 'a' To 'f'
              num = num | (*c\c - 'a' + 10)
            Case 'A' To 'F'
              num = num | (*c\c - 'A' + 10)
          EndSelect
          num << 4
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          Select *c\c
            Case '0' To '9'
              num = num | (*c\c - '0')
            Case 'a' To 'f'
              num = num | (*c\c - 'a' + 10)
            Case 'A' To 'F'
              num = num | (*c\c - 'A' + 10)
          EndSelect
          num << 4
          *C + SizeOf(character):If *C\c = 0 : Break : EndIf
          Select *c\c
            Case '0' To '9'
              num = num | (*c\c - '0')
            Case 'a' To 'f'
              num = num | (*c\c - 'a' + 10)
            Case 'A' To 'F'
              num = num | (*c\c - 'A' + 10)
          EndSelect
          *Z\c = num
        Case 0
          Break
        Default
          *Z\c = *c\c
      EndSelect
      
    Else
      *Z\c = *C\c
    EndIf
    *C + SizeOf(Character)
    *Z + SizeOf(Character)
  Wend
  
  ProcedureReturn result$
EndProcedure

; UnescapeString() with user-defined lookup function (searching for ${abcdefgh} variables)
Procedure$ ParseWildcards(String$, LookUp.UE_LookupProcedure, *udata, iswriter = #False)
  Protected resultLength
  Protected *C.Character = @String$
  Protected t$, *s, enca, lims, lime
  Protected tlen, i
  Protected NewList token.s()
  While *C\c
    If *C\c = '$'   ; dollar sign (will be omitted)
      *C + SizeOf(Character)
      Select *C\c
        Case 'a' To 'z', 'A' To 'Z', '_'
          tlen = 0
          *S   = *C
          While *C\c
            Select *C\c
              Case 'a' To 'z', 'A' To 'Z', '_', '0' To '9'
                tlen + 1
              Default
                Break
            EndSelect
            *C + SizeOf(character)
          Wend
          AddElement(Token())
          t$      = LookUp(*udata, PeekS(*S, tlen), 0, isWriter)
          Token() = t$
          resultLength + Len(t$)
          Continue
          
        Case '{', '[', '('
          lims = *c\c
          If lims = '('
            lime = lims + 1
          Else
            lime = lims + 2
          EndIf
          *C + SizeOf(character)
          *S   = *C
          enca = 0
          tlen = 0
          While *C\c
            If (*C\c = 34 Or *C\c = 39)
              If enca = *C\c
                enca = 0
              ElseIf enca = 0
                enca = *C\c
              EndIf
            ElseIf *C\c = '\'
              resultLength + 1
              *C + SizeOf(Character)
              If Not *C\c
                Break
              EndIf
              Continue
            ElseIf *C\c = lime
              If enca = 0
                Break
              EndIf
            EndIf
            tlen + 1
            *C + SizeOf(Character)
          Wend
          
          If *C\c = lime
            AddElement(Token())
            t$      = LookUp(*udata, PeekS(*S, tlen), lims, isWriter)
            Token() = t$
            resultLength + Len(t$)
            *C + SizeOf(Character)
            Continue
          Else
            Break
          EndIf
          
          
        Default
          
      EndSelect
      
    EndIf
    resultLength + 1 ; next character will be copied (no matter what it is) -> best choice $
    *C + SizeOf(Character)
  Wend
  
  Protected result$ = Space(resultLength)
  Protected *Z.Character = @result$
  Protected num.c
  
  *C = @String$
  
  While *C\c
    If *C\c = '$'
      *C + SizeOf(Character)
      Select *C\c
        Case 'a' To 'z', 'A' To 'Z', '_'
          While *C\c
            Select *C\c
              Case 'a' To 'z', 'A' To 'Z', '_', '0' To '9'
              Default
                Break
            EndSelect
            *C + SizeOf(character)
          Wend
          FirstElement(Token())
          *Z + PokeS(*Z, token())
          DeleteElement(token())
          Continue
          
        Case '{', '(', '['
          lims = *C\c
          If lims = '('
            lime = lims + 1
          Else
            lime = lims + 2
          EndIf
          
          While *C\c
            If (*C\c = 34 Or *C\c = 39)
              If enca = *C\c
                enca = 0
              ElseIf enca = 0
                enca = *C\c
              EndIf
            ElseIf *C\c = '\'
              *Z\c = *C\c
              *C + SizeOf(Character)
              If Not *C\c
                Break
              EndIf
              *Z\c = *C\c
              *C + SizeOf(Character)
              *S = *C
              Continue
              
            ElseIf *C\c = lime
              If enca = 0
                Break
              EndIf
            EndIf
            *C + SizeOf(Character)
          Wend
          
          If *C\c = lime
            FirstElement(Token())
            
            *Z + PokeS(*Z, Token())
            DeleteElement(token())
            
            *c + SizeOf(Character)
            Continue
          EndIf
          Break
        Default
          *Z\c = *C\c
      EndSelect
      
    Else
      *Z\c = *C\c
    EndIf
    *C + SizeOf(Character)
    *Z + SizeOf(Character)
  Wend
  ProcedureReturn result$
EndProcedure

; UnescapeString() with user-defined lookup and output/writer function
Procedure.i ParseWildcards_Back(String$, *udataWriter, *udataLookup, Writer.UE_WriterProcedure, Lookup.UE_LookupProcedure)
  
  Structure UE_Token
    Token.s
    toklen.i
  EndStructure
  
  Protected *C.Character = @String$
  Protected t$, *s, num.c
  Protected result , enca, lims, lime
  Protected toklen, ttlen, x$, c, *Z
  Protected NewList token.UE_Token()
  *S = *C
  
  
  Macro WR(_X_)
    AddElement(Token())
    Token()\token  = _X_
    Token()\toklen = Len(_X_)
    ttlen          = ttlen + token()\toklen
    toklen         = 0
    *S             = *C + SizeOf(Character)
  EndMacro
  
  Macro WR1(_X_)
    AddElement(Token())
    Token()\token  = _X_
    Token()\toklen = 1
    ttlen + 1
    toklen = 0
    *S     = *C + SizeOf(Character)
  EndMacro
  
  Macro WRFLUSH()
    c  = 0
    x$ = Space(ttlen + 2)
    *z = @x$
    ForEach token()
      c + PokeS(*z + c, token()\token)
    Next
    ClearList(token())
    ttlen  = 0
    toklen = 0
    *S     = *C + SizeOf(Character)
    If c
      result = Writer(*udataWriter, *z, c)
      If result : ProcedureReturn result : EndIf
    EndIf
  EndMacro
  
  While *C\c
    If toklen = 0
      *S = *C
    EndIf
    If *C\c = '$'   ; dollar sign (will be omitted)
      If toklen
        WR(PeekS(*S, toklen))
      EndIf
      *C + SizeOf(Character)
      Select *C\c
        Case 'a' To 'z', 'A' To 'Z', '_'
          WRFLUSH()
          *S     = *C
          toklen = 0
          While *C\c
            Select *C\c
              Case 'a' To 'z', 'A' To 'Z', '_', '0' To '9'
                toklen + 1
              Default
                Break
            EndSelect
            *C + SizeOf(Character)
          Wend
          If toklen
            t$     = Lookup(*udataLookup, PeekS(*S, toklen), 0, 1)
            toklen = 0
            WR(t$)
          EndIf
          *S     = *C
          toklen = 0
          Continue
          
        Case '{', '[', '(' ; look up procedure
          WRFLUSH()
          lims = *C\c
          If lims = '(' : lime = lims + 1 : Else : lime = lims + 2 : EndIf
          *C + SizeOf(character)
          *S   = *C ;+ SizeOf(Character)
          enca = 0
          While *C\c
            If enca And enca = *C\c
              enca = 0
              
            ElseIf *c\c = 34 Or *C\c = 39 And enca = 0
              enca = *C\c
              
            ElseIf *C\c = '\'
              *C + SizeOf(Character)
              If Not *C\c
                Break
              EndIf
            ElseIf enca = 0 And *C\c = lime
              Break
            EndIf
            toklen + 1
            *C + SizeOf(Character)
          Wend
          
          If *C\c = lime
            t$ = LookUp(*udataLookup, PeekS(*S, toklen), lims, 1)
            If t$ <> #Empty$ And T$ <> #Empty$
              toklen = 0
              WR(t$)
            EndIf
            toklen = 0
            *C + SizeOf(Character)
            *S = *C
            Continue
          Else
            Break
          EndIf
          toklen = 0
          *S     = *C + SizeOf(Character)
        Default
          WR1(Chr(*C\c))
      EndSelect
      
    Else
      toklen + 1
    EndIf
    *C + SizeOf(Character)
    
  Wend
  If toklen
   ; Debug #PB_Compiler_Procedure +" exit tail peek call"
    WR(PeekS(*S, toklen))
  EndIf
  
  WRFLUSH()
 ; Debug #PB_Compiler_Procedure +" exit bottom"
  ProcedureReturn 0
EndProcedure

; Explodes a string like if it was a command/batch line taking care of " ' parameter encapsulation
Procedure ExplodeArgString(String$, Array Result.s(1)) ;
  Protected *C.Character
  Protected count, c
  Protected *argstart, slen, argend, *n.Character
  *C = @String$
  While *C\c
    If (*C\c = 32 Or *C\c = #TAB) And *argstart
      *argstart = 0
      slen      = 0
      count + 1
    EndIf
    While *C\c = 32 Or *C\c = #TAB : *C + SizeOf(Character) : Wend
    Select *C\c
      Case 0, '#' ; commentary     ; End of string
        Break
      Case '\'
        If *argstart = 0
          *argstart = *c
          *C + SizeOf(Character)
          If *c\c = 0
            Break
          EndIf
          slen = 2
          *C + SizeOf(Character)
          Continue
        Else
          slen + 1
        EndIf
      Case 34, 39 ; Encapsulation
        count + 1
        argend    = *C\c
        *argstart = *C + SizeOf(Character)
        slen      = 0
        *C + SizeOf(Character)
        While *C\c
          If *C\c = '\'
            slen + 1
            *C + SizeOf(Character)
            If *C\c = 0
              Break
            EndIf
          ElseIf *C\c = argend
            ;slen+1
            Break
          EndIf
          slen + 1
          *C + SizeOf(Character)
        Wend
        slen      = 0
        *argstart = 0
      Default
        If *argstart = 0
          *argstart = *C
          slen      = 1
          count     = count + 1
        Else
          slen = slen + 1
        EndIf
    EndSelect
    If *C\c
      *C = *C + SizeOf(Character)
    Else
      Break
    EndIf
  Wend
  If slen
    count = count + 1
  EndIf
  ReDim Result.s(count)
  *argstart = 0
  *C        = @String$
  c         = 0
  While *C\c
    If (*C\c = 32 Or *c\c = #TAB) And *argstart
      If slen
        result(c) = PeekS(*argstart, slen)
      Else
        result(c) = ""
      EndIf
      c         = c + 1
      slen      = 0
      *argstart = 0
    EndIf
    While *C\c = 32 Or *C\c = #TAB : *C + SizeOf(Character) : Wend
    Select *C\c
      Case 0, '#' ; commentary     ; End of string
        Break
      Case '\'
        If *argstart = 0
          *argstart = *c
          *C + SizeOf(Character)
          If *c\c = 0
            Break
          EndIf
          slen = 2
          *C + SizeOf(Character)
          Continue
        Else
          slen + 1
        EndIf
      Case 34, 39 ; Encapsulation
        argend    = *C\c
        *argstart = *C + SizeOf(Character)
        slen      = 0
        *C + SizeOf(Character)
        While *c\c
          If *c\c = '\'
            *C   = *c + SizeOf(Character)
            slen = slen + 1
            If *C\c = 0
              Break
            EndIf
          ElseIf *C\c = argend
            ;slen+1
            Break
          EndIf
          slen = slen + 1
          *C + SizeOf(Character)
        Wend
        If slen
          result(c) = PeekS(*argstart, slen)
        Else
          result(c) = ""
        EndIf
        *argstart = 0
        slen      = 0
        c         = c + 1
      Default
        If *argstart = 0
          *argstart = *C
          slen      = 1
        Else
          slen = slen + 1
        EndIf
    EndSelect
    
    If *C\c
      *C + SizeOf(Character)
    Else
      Break
    EndIf
  Wend
  If slen
    result(c) = PeekS(*argstart, slen)
    c         = c + 1
  EndIf
  
  ReDim Result.s(c)
  ProcedureReturn c
EndProcedure

; Like EplodeArgString() except that the results are parsed with a lookup procedure
Procedure ExplodeArgString2(String$, Array Result.s(1), LookUp.UE_LookupProcedure, *udata, isWriter) ;
  Protected *C.Character
  Protected count, c
  Protected *argstart.character, slen, argend, *n.Character
  
  ;Protected old_show=show_it
  ;show_it=#True
  
  *C = @String$
  While *C\c
    If (*C\c = 32 Or *C\c = #TAB) And *argstart
      *argstart = 0
      slen      = 0
      count + 1
    EndIf
    While *C\c = 32 Or *C\c = #TAB : *C + SizeOf(Character) : Wend
    Select *C\c
      Case 0, '#' ; commentary     ; End of string
        Break
      Case '\'
        If *argstart = 0
          *argstart = *c
          *C + SizeOf(Character)
          If *c\c = 0
            Break
          EndIf
          slen = 2
          *C + SizeOf(Character)
          Continue
        Else
          slen + 1
        EndIf
      Case 34, 39 ; Encapsulation
        count + 1
        argend    = *C\c
        *argstart = *C ;+ SizeOf(Character)
        slen      = 1
        *C + SizeOf(Character)
        While *C\c
          If *C\c = '\'
            slen + 1
            *C + SizeOf(Character)
            If *C\c = 0
              Break
            EndIf
          ElseIf *C\c = argend
            slen + 1
            Break
          EndIf
          slen + 1
          *C + SizeOf(Character)
        Wend
        slen      = 0
        *argstart = 0
      Default
        If *argstart = 0
          *argstart = *C
          slen      = 1
          count     = count + 1
        Else
          slen = slen + 1
        EndIf
    EndSelect
    If *C\c
      *C = *C + SizeOf(Character)
    Else
      Break
    EndIf
  Wend
  If slen
    count = count + 1
  EndIf
  ReDim Result.s(count)
  *argstart = 0
  *C        = @String$
  c         = 0
  While *C\c
    If (*C\c = 32 Or *c\c = #TAB) And *argstart
      If slen
        result(c) = PeekS(*argstart, slen)
      Else
        result(c) = ""
      EndIf
      c         = c + 1
      slen      = 0
      *argstart = 0
    EndIf
    While *C\c = 32 Or *C\c = #TAB : *C + SizeOf(Character) : Wend
    Select *C\c
      Case 0, '#' ; commentary     ; End of string
        Break
      Case '\'
        If *argstart = 0
          *argstart = *c
          *C + SizeOf(Character)
          If *c\c = 0
            Break
          EndIf
          slen = 2
          *C + SizeOf(Character)
          Continue
        Else
          slen + 1
        EndIf
      Case 34, 39 ; Encapsulation
        argend    = *C\c
        *argstart = *C + SizeOf(Character)
        slen      = 0
        *C + SizeOf(Character)
        While *c\c
          If *c\c = '\'
            *C   = *c + SizeOf(Character)
            slen = slen + 1
            If *C\c = 0
              Break
            EndIf
          ElseIf *C\c = argend
            ;slen+1
            Break
          EndIf
          slen = slen + 1
          *C + SizeOf(Character)
        Wend
        If slen
          If *argstart\c = 39
            result(c) = ParseWildcards(PeekS(*argstart + SizeOf(Character), slen - (2*SizeOf(Character))), LookUp, *udata, isWriter)
          ElseIf *argstart\c = 34
            result(c) = ParseWildcards(PeekS(*argstart + SizeOf(Character), slen - (2*SizeOf(Character))), LookUp, *udata, isWriter)
          Else
            result(c) = ParseWildcards(PeekS(*argstart, slen), LookUp, *udata, isWriter)
          EndIf
        Else
          result(c) = ""
        EndIf
        *argstart = 0
        slen      = 0
        c         = c + 1
      Default
        If *argstart = 0
          *argstart = *C
          slen      = 1
        Else
          slen = slen + 1
        EndIf
    EndSelect
    
    If *C\c
      *C + SizeOf(Character)
    Else
      Break
    EndIf
  Wend
  If slen
    If *argstart\c = 39
      result(c) = ParseWildcards(PeekS(*argstart + SizeOf(Character), slen - (2*SizeOf(Character))), LookUp, *udata, isWriter)
    ElseIf *argstart\c = 34
      result(c) = ParseWildcards(PeekS(*argstart + SizeOf(Character), slen - (2*SizeOf(Character))), LookUp, *udata, isWriter)
    Else
      result(c) = ParseWildcards(PeekS(*argstart, slen), LookUp, *udata, isWriter)
    EndIf
    c = c + 1
  EndIf
  
  ReDim Result.s(c)
  ProcedureReturn c
EndProcedure

; returns the named flag's value (flag1:val1 flag2:val2) values are in decimal
Procedure.i StringFlag(String$, FlagList$)
  Protected *S
  Protected klen
  Protected flag
  Protected value
  Protected *C.character = @flaglist$
  
  While *C\c
    While *C\c = #TAB Or *C\c = 32
      *C + SizeOf(character)
    Wend
    *S   = *C
    klen = 0
    While *C\c And *C\c <> 32 And *c\c <> #TAB
      If *C\c = ':'
        If PeekS(*S, klen) = String$
          ;Debug "found! "+String$
          ; continue
          *C + SizeOf(Character)
          While *C\c And *C\c <> 32 And *C\c <> #TAB
            value * 10
            value + (*C\c - '0')
            *C + SizeOf(Character)
          Wend
          ProcedureReturn value
        Else
         ; Debug "next... ("+PeekS(*S,klen)+")"
          While *C\c And *c\c <> 32 And *C\c <> #TAB
            *C + SizeOf(character)
          Wend
          *S = *C
          If Not *C\c
            ;Debug "reached eol"
            ProcedureReturn
          EndIf
          Break
        EndIf
      EndIf
      klen + 1
      *C + SizeOf(Character)
    Wend
    *C + SizeOf(Character)
  Wend
  ;Debug "not found :("
  ProcedureReturn flag
EndProcedure

; returns #true is string$ is in the memberlist
Procedure.i IsMember(String$, MemberList$)
  Protected *Z1.Character
  Protected *Z2.character = @MemberList$
  While *Z2\c
    While *Z2\c = #TAB Or *Z2\c = 32
      *Z2 + SizeOf(Character)
    Wend
    *Z1 = @String$
    While *Z1\c = *Z2\c And *Z1\c And *Z2\c
      *Z1 + SizeOf(Character)
      *Z2 + SizeOf(Character)
    Wend
    If *Z1\c = 0 And (*Z2\c = 0 Or *Z2\c = 32 Or *Z2\c = #TAB)
      ProcedureReturn #True
    EndIf
  Wend
  ProcedureReturn #False
EndProcedure

CompilerIf #PB_Compiler_IsMainFile
  
; Examples
; --------------------------------------------------------------------------------------------------------
; How to make your own lookup function to return unescaped+filled strings
; Use helpfer functions like ExplodeArgString and StringFlag
;
; use StringFlag(String$,) with flaglists like these:
  
  #INCLUDE_MSBOX = #True
  #USE_CONSOLE   = #True
  Global msgboxflaglist$ ; pairs are name:value, pairs are separated by space(s) or tab(s)
  msgboxflaglist$ = "cancel:" + Str(#PB_MessageRequester_Cancel)
  msgboxflaglist$ + " error:" + Str(#PB_MessageRequester_Error)
  msgboxflaglist$ + " info:" + Str(#PB_MessageRequester_Info)
  msgboxflaglist$ + " no:" + Str(#PB_MessageRequester_No)
  msgboxflaglist$ + " ok:" + Str(#PB_MessageRequester_Ok)
  msgboxflaglist$ + " warning:" + Str(#PB_MessageRequester_Warning)
  msgboxflaglist$ + " yes:" + Str(#PB_MessageRequester_Yes)
  msgboxflaglist$ + " yes-no-cancel:" + Str(#PB_MessageRequester_YesNoCancel)
  msgboxflaglist$ + " yes-no:" + Str(#PB_MessageRequester_YesNo)
  
; Example 1:
; look up procedures receive "CONTENT" key$ when found inside parsed/unescaped string "${CONTENT}"
  NewMap _L.s()
  
  Declare$ prime_lookup(*udata, key$, parenthesis, isWriter)
  
  Procedure$ mylookup(*udata, key$, parenthesis, isWriter)
    Shared _L()
    Protected result$ = ""
    Protected Dim p$(8)
   ; Protected pc = ExplodeArgString2(key$, p$(), @mylookup(), *udata, isWriter)
    Protected pc = ExplodeArgString2(key$, p$(), @prime_lookup(), *udata, Bool(isWriter) * 2)
    ;Debug key$
    If pc < 6
      ReDim p$(6)
    EndIf
    
    If parenthesis = 0
      Select LCase(p$(0))
        Case "username"
          result$ = UserName()
        Case "computername"
          result$ = ComputerName()
        Default
          result$ = GetRuntimeString(p$(0))
      EndSelect
      ProcedureReturn result$
    ElseIf parenthesis =  '['
      ; lang
      result$ = _L(p$(0))
      ProcedureReturn result$
    EndIf
    
    Select p$(0)
      Case "n" ; CRLF 'Telnet' 'network-line' standard ${n} may be easier to type than \r\n for some !?
        result$ = #CRLF$
      Case "env"
        If pc = 2
          result$ = GetEnvironmentVariable(p$(1))
        EndIf
      Case "rts"
        result$ = GetRuntimeString(p$(1))
      Case "rti"
        result$ = FormatNumber(GetRuntimeInteger(p$(1)), 0, "", p$(2))
      Case "rtd"
        result$ = FormatNumber(GetRuntimeDouble(p$(1)), Val(p$(2)), Left(p$(3) + ",", 1), Mid(p$(3), 2, 1))
      Case "setrts"
        SetRuntimeString(p$(1), p$(2))
      Case "setrti"
        SetRuntimeInteger(p$(1), Val(p$(2)))
      Case "setrtd"
        SetRuntimeDouble(p$(1), ValD(p$(2)))
      Case "lset"
        result$ = LSet(p$(1), Val(p$(2)), p$(3))
      Case "rset"
        result$ = RSet(p$(1), Val(p$(2)), p$(3))
      Case "cset"
        P$(1)   = ParseWildcards(p$(1), @prime_lookup(), *udata, 0)
        result$ = LSet(Space((Val(p$(2)) / 2) - (Len(p$(1)) / 2)), Val(p$(2)), P$(3))
      Case "mid"
        result$ = Mid(p$(1), Val(p$(2)), Val(p$(3)))
      Case "rf"
        Protected fh = ReadFile( - 1, p$(1))
        Protected format, remove_eol, replace_eol
        If fh
          format  = ReadStringFormat(fh)
          result$ = ReadString(fh, format | #PB_File_IgnoreEOL)
          CloseFile(fh)
          Select p$(2)
            Case ""
            Case "remove-eol"
              Repeat
                Select Right(result$, 1)
                  Case #CR$, #LF$
                    result$ = Left(result$, Len(result$) - 1)
                  Default
                    Break
                EndSelect
              ForEver
            Case "replace-eol"
              result$ = ReplaceString(result$, #CRLF$, " ")
              result$ = ReplaceString(result$, #LFCR$, " ")
              result$ = ReplaceString(result$, #LF$, " ")
              result$ = ReplaceString(result$, #CR$, " ")
              result$ = ReplaceString(result$, #TAB$, LSet("", 8))
              result$ = Trim(result$)
          EndSelect
          
        EndIf
        CompilerIf #INCLUDE_MSBOX
        Case "msgbox"
      ;Debug ">>"+key$
          If pc => 4
            Select MessageRequester(ParseWildcards(p$(1), @mylookup(), *udata), ParseWildcards(p$(2), @mylookup(), *udata), StringFlag(p$(3), msgboxflaglist$) | StringFlag(p$(4), msgboxflaglist$))
              Case #PB_MessageRequester_Cancel
                result$ = "cancel"
              Case #PB_MessageRequester_No
                result$ = "no"
              Case #PB_MessageRequester_Yes
                result$ = "yes"
            EndSelect
          ElseIf pc >= 3
            MessageRequester(ParseWildcards(p$(1), @mylookup(), *udata), ParseWildcards(p$(2), @mylookup(), *udata))
          ElseIf pc => 2
            MessageRequester("Nested msgbox", ParseWildcards(p$(1), @mylookup(), *udata))
          EndIf
        CompilerEndIf
    EndSelect
    ProcedureReturn result$
  EndProcedure
  
  Procedure$ MyUnescapeString(String$)
    EnableGraphicalConsole(0)
    ProcedureReturn _UnescapeString(ParseWildcards(String$, @prime_lookup(), 0))
  EndProcedure

 ; msgbox in mylookup() needs to be uncommented For it To work
  ;MessageRequester("Too much??", MyUnescapeString("Hello world\r\nSomething wrong?\r\n${msgbox 'Heh heh heh :)' 'A msgbox hiding in a msgbox!!' yes-no-cancel warning}!!!"), #PB_MessageRequester_YesNoCancel)
  
; --------------------------------------------------------------------------------------------------------
; How to make your own output writer
; How to make it work with the lookup function
; How to simply cascade lookup functions with select statements
  
  Define CurrentConsoleINV
  Define CurrentConsoleBG = 0, CurrentConsoleFG = 7
  Declare _Print(String$)
  
  Procedure$ Lang(id$)
    Shared _L()
    If FindMapElement(_L(), id$)
      ProcedureReturn _L()
    EndIf
    ProcedureReturn ""
  EndProcedure
  
  Procedure$ PrimeParse(String$, isWriter = 0)
    Protected result$ = ParseWildcards(String$, @prime_lookup(), 0, isWriter)
    ProcedureReturn result$
  EndProcedure
  
  Procedure$ prime_lookup(*udata, key$, parenthesis, isWriter)
    Shared CurrentConsoleFG
    Shared CurrentConsoleBG
    Shared CurrentConsoleINV
    
    Protected Dim p$(0)
    Protected pc = ExplodeArgString2(key$, p$(), @prime_lookup(), *udata, Bool(isWriter) * 2)
    Protected result$ = Space(100)
    Protected Raw
    Protected inkey$
    result$ = #Empty$
    ReDim p$(8)
    
  ; Graphical console functions
    Select LCase(P$(0))
        
      Case "fg"
        
        If isWriter = 1
          CompilerIf #USE_CONSOLE
            EnableGraphicalConsole(1)
            CurrentConsoleFG = Val(p$(1))
            If CurrentConsoleINV = 0
              ConsoleColor(CurrentConsoleFG, CurrentConsoleBG)
            Else
              ConsoleColor(CurrentConsoleBG, CurrentConsoleFG)
            EndIf
            
            EnableGraphicalConsole(0)
          CompilerEndIf
        ElseIf isWriter = 2
          result$ = "${" + key$ + "}"
        EndIf
        
        
      Case "bg"
        
        If isWriter = 1
          CompilerIf #USE_CONSOLE
            EnableGraphicalConsole(1)
            CurrentConsoleBG = Val(p$(1))
            If CurrentConsoleINV = 0
              ConsoleColor(CurrentConsoleFG, CurrentConsoleBG)
            Else
              ConsoleColor(CurrentConsoleBG, CurrentConsoleFG)
            EndIf
            EnableGraphicalConsole(0)
          CompilerEndIf
        ElseIf isWriter = 2
          result$ = "${" + key$ + "}"
        EndIf
        
      Case "cls"
        If isWriter = 1
          ;ClearConsole()
          CurrentConsoleBG = 0
          CurrentConsoleFG = 7
          CompilerIf #USE_CONSOLE
            EnableGraphicalConsole(1)
            
            If CurrentConsoleINV = 0
              ConsoleColor(CurrentConsoleFG, CurrentConsoleBG)
            Else
              ConsoleColor(CurrentConsoleBG, CurrentConsoleFG)
            EndIf
            
            EnableGraphicalConsole(0)
          CompilerEndIf
        ElseIf isWriter = 2
          result$ = "${" + key$ + "}"
        EndIf
        
        
      Case "col"
        
        If isWriter = 1
          CurrentConsoleFG = Val(p$(1))
          CurrentConsoleBG = Val(p$(2))
          CompilerIf #USE_CONSOLE
            EnableGraphicalConsole(1)
            
            If CurrentConsoleINV = 0
              ConsoleColor(CurrentConsoleFG, CurrentConsoleBG)
            Else
              ConsoleColor(CurrentConsoleBG, CurrentConsoleFG)
            EndIf
            
            EnableGraphicalConsole(0)
          CompilerEndIf
        ElseIf isWriter = 2
          result$ = "${" + key$ + "}"
        EndIf
        
        
      Case "n"
        result$ = #CRLF$
        
      Case "h"
        
        If isWriter = 1
          CompilerIf #USE_CONSOLE
            EnableGraphicalConsole(1)
            CompilerIf #PB_Compiler_OS = #PB_OS_Linux
              ConsoleLocate(1, 1)
            CompilerElse
              ConsoleLocate(0, 0)
            CompilerEndIf
            EnableGraphicalConsole(0)
          CompilerEndIf
        ElseIf isWriter = 2
          result$ = "${" + key$ + "}"
        EndIf
        
        
      Case "inv"
        
        If isWriter = 1
          CurrentConsoleINV ! 1
          CompilerIf #USE_CONSOLE
            EnableGraphicalConsole(1)
            If CurrentConsoleINV = 0
              ConsoleColor(CurrentConsoleFG, CurrentConsoleBG)
            Else
              ConsoleColor(CurrentConsoleBG, CurrentConsoleFG)
            EndIf
            EnableGraphicalConsole(0)
          CompilerEndIf
        EndIf
        
        
      Case "cur"
        
        If isWriter = 1
          If pc = 1
            CompilerIf #USE_CONSOLE
              ConsoleCursor(Val(p$(1)))
            CompilerEndIf
          EndIf
        EndIf
        
        
      Case "loc"
        
        If isWriter = 1
          If pc = 2
            CompilerIf #USE_CONSOLE
              EnableGraphicalConsole(1)
              CompilerIf #PB_Compiler_OS = #PB_OS_Linux
                ConsoleLocate(Val(p$(0)), Val(p$(1)))
              CompilerElse
                ConsoleLocate(Val(p$(0)), Val(p$(1)))
              CompilerEndIf
              EnableGraphicalConsole(0)
            CompilerEndIf
          EndIf
        ElseIf isWriter = 2
          result$ = "${" + key$ + "}"
        EndIf
        
      Case "rep"
        
        result$ = LSet("", Val(p$(1)), p$(2))
        
        
        
      Case "reset", "r"
        
        If isWriter = 1
          CompilerIf #USE_CONSOLE
            EnableGraphicalConsole(1)
            CurrentConsoleBG  = 0
            CurrentConsoleFG  = 7
            CurrentConsoleINV = 0
            ConsoleColor(7, 0)
            EnableGraphicalConsole(0)
          CompilerEndIf
        ElseIf isWriter = 2
          result$ = "${" + key$ + "}"
        EndIf
        
        
      Case "hd", "head"
        
        Protected tt$ = ParseWildcards(p$(1), @prime_lookup(), 0, 0)
        tt$   = Left(tt$, 76)
        p$(1) = _UnescapeString(P$(1))
        p$(2) = _UnescapeString(P$(2))
        p$(3) = _UnescapeString(P$(3))
        p$(4) = _UnescapeString(P$(4))
        Protected ind$ = Space(Val(p$(4)))
        Protected tlen = Len(tt$)
        Protected horiz$ = Left(p$(2), 1)
        
        If horiz$ = "" : horiz$ = "*" : EndIf
        
        Protected vert$ = Mid(p$(2), 2, 1)
        If vert$ = "" : vert$ = horiz$ : EndIf
        
        Protected tl$ = Left(p$(3), 1)
        If tl$ = "" : If vert$ = horiz$ : tl$ = horiz$:Else:tl$ = "+":EndIf :EndIf
        
        Protected tr$ = Mid(p$(3), 2, 1)
        If tr$ = "":tr$ = tl$:EndIf
        
        Protected bl$ = Mid(p$(3), 3, 1)
        If bl$ = "":bl$ = tl$:EndIf
        
        Protected br$ = Mid(p$(3), 4, 1)
        If br$ = "":br$ = tl$:EndIf
        
        If isWriter = 0
          result$ =ind$ +  tl$ + LSet("", tlen + 2, horiz$) + tr$ + #CRLF$ + ind$ + vert$ + " " + tt$ + " " + vert$ + #CRLF$ + ind$ +  bl$ + LSet("", tlen + 2, horiz$) + br$ + #CRLF$
        ElseIf isWriter = 2
          result$ = ind$ + tl$ + LSet("", tlen + 2, horiz$) + tr$ + #CRLF$ + ind$ + vert$ + " " + _UnescapeString(ParseWildcards(p$(1), @prime_lookup(), 0, 2)) + " " + vert$ + #CRLF$ +ind$ + bl$ + LSet("", tlen + 2, horiz$) + br$ + #CRLF$
        Else
          CompilerIf #USE_CONSOLE
            _Print(ind$ + tl$ + LSet("", tlen + 2, horiz$) + tr$ + #CRLF$)
            _Print(ind$ + vert$ + " " + p$(1) + " " + vert$ + #CRLF$)
            _Print(ind$ + bl$ + LSet("", tlen + 2, horiz$) + br$ + #CRLF$)
          CompilerEndIf
          result$ = ""
        EndIf
        
      Case "inln"
        
        CompilerIf #USE_CONSOLE
          If isWriter = 1
            Protected input_len = Val(p$(1))
            Protected fill_char$ = Left(p$(3) + " ", 1)
            Protected value$ = p$(2)
            Protected inloc
            Protected escape_sequence$
            
            EnableGraphicalConsole(0)
            Print(LSet(value$, input_len, fill_char$))
            Print(LSet("", input_len - Len(value$), Chr(8)))
            inloc = Len(value$)
            
            Repeat
              EnableGraphicalConsole(1)
              inkey$ = Inkey()
              raw    = RawKey()
              
              Select raw
                Case 0
                  Delay(0)
                Case 8, 127
                  If Len(value$) > 0 And inloc > 0
                    
                    EnableGraphicalConsole(0)
                    
                    value$ = Left(value$, inloc - 1) + Mid(value$, inloc + 1)
                    Print(Chr(8) + fill_char$ + Chr(8))
                    Print(Mid(value$, inloc) + fill_char$)
                    
                    Print(LSet("", 2 + Len(value$) - inloc, Chr(8)))
                    inloc - 1
                    
                ;
                    
                    
                  EndIf
                  
                Case 13, 10
                  result$ = value$
                  Break
                Case 27
                  If Len(inkey$) < 2
                    While Len(value$)
                      value$ = Left(value$, Len(value$) - 1)
                      Print(Chr(8) + fill_char$ + Chr(8))
                    Wend
                    inloc = 0
                  EndIf
                  
                Case 37
               ; Debug "left"
                  If inloc > 0
                    inloc - 1
                    Print(Chr(8))
                  EndIf
                Case 38
              ;  Debug "up"
                Case 39
               ; Debug "right"
                  If inloc < Len(value$)
                    inloc + 1
                    Print(Mid(value$, inloc, 1))
                  EndIf
                Case 40
                ;  Debug "down"
                Case 46
                ; delete key
                  If Len(value$) > 0
                    If inloc < Len(value$)
                      value$ = Left(value$, inloc) + Mid(value$, inloc + 2)
                      Print(Mid(value$, inloc + 1) + fill_char$)
                      Print(LSet("", 1 + Len(Mid(value$, inloc + 1)), Chr(8)))
                    EndIf
                  EndIf
                  
                Default
                  Debug "default... raw: " + Raw
                  
                  
              EndSelect
              If raw >= 32 And Len(inkey$) = 1
                If Len(value$) < input_len
                  value$ = Left(value$, inloc) + inkey$ + Mid(value$, inloc + 1)
                  inloc + 1
                  
                  Print(Mid(value$, inloc))
                  Print(LSet("", Len(value$) - inloc, Chr(8)))
                  
                  
                EndIf
                
                
              ElseIf raw = 27
                
              EndIf
            ForEver
         ; Debug "value$="
         ; Debug value$
            
            If IsRuntime(p$(4))
              SetRuntimeString(p$(4), value$)
              result$ = ""
            Else
              result$ = value$
            EndIf
          EndIf
          
        CompilerEndIf
        
      Case "inpw"
        CompilerIf #USE_CONSOLE
          If isWriter = 1
            input_len  = Val(p$(1))
            fill_char$ = Left(p$(3) + " ", 1)
            value$     = p$(2)
            Protected pw_char$ = p$(4)
            
            If pw_char$ = ""
              pw_char$ = "*"
            ElseIf pw_char$ = "no-char"
              pw_char$ = ""
            EndIf
            
            inloc = Len(value$)
            EnableGraphicalConsole(0)
            If value$ <> "" And pw_char$ <> ""
              Print(LSet(LSet("", Len(value$), pw_char$), input_len, fill_char$))
              Print(LSet("", input_len - Len(value$), Chr(8)))
              
            Else
              Print(LSet("", input_len, fill_char$))
              Print(LSet("", input_len, Chr(8)))
            EndIf
            Repeat
              inkey$ = Inkey()
              raw    = RawKey()
              
              Select raw
                Case 0
                  Delay(0)
                Case 10, 13
                  
                  result$ = value$
                  Break
                  
                Case 27
                  If pw_char$ <> ""
                    While Len(value$)
                      value$ = Left(value$, Len(value$) - 1)
                      Print(Chr(8) + fill_char$ + Chr(8))
                    Wend
                  EndIf
                  inloc = 0
                  
                Case 37
                  
                  If inloc > 0
                    inloc - 1
                    If pw_char$ <> ""
                      Print(Chr(8))
                    EndIf
                  EndIf
                Case 38
                  
                Case 39
                  If inloc < Len(value$)
                    inloc + 1
                    Print(pw_char$)
                  EndIf
                Case 40
                  
              EndSelect
              
              
              If raw >= 32 And inkey$ <> ""
                If Len(value$) < input_len
                  value$ = Left(value$, inloc) + inkey$ + Mid(value$, inloc + 1)
                  inloc + 1
                  If pw_char$ <> ""
                    Print(LSet("", 1 + Len(value$) - inloc, pw_char$))
                    Print(LSet("", Len(value$) - inloc, Chr(8)))
                  EndIf
                EndIf
                
              ElseIf raw = 8
                If Len(value$) > 0 And inloc > 0
                  value$ = Left(value$, Len(value$) - 1)
                  If pw_char$ <> ""
                    Print(Chr(8) + fill_char$ + Chr(8))
                  EndIf
                  inloc - 1
                  If pw_char$ <> ""
                    Print(LSet("", 1 + Len(value$) - inloc, pw_char$))
                    Print(LSet("", Len(value$) - inloc, Chr(8)))
                  EndIf
                EndIf
                
              EndIf
            ForEver
            
            If IsRuntime(p$(5))
              SetRuntimeString(p$(5), value$)
              result$ = ""
            Else
              result$ = value$
            EndIf
            
          EndIf
        CompilerEndIf
        
      Case "waitkey"
        Protected kl = Len(p$(1))
        CompilerIf #USE_CONSOLE
          If isWriter = 1
            If kl = 0
              Repeat
                result$ = Inkey()
                Raw     = RawKey()
                If Not Raw
                  Delay(0)
                EndIf
              Until result$ <> #Empty$ Or raw
              result$ = result$ + ":" + Str(raw)
              
            Else
              
              Repeat
                result$ = Inkey()
                Raw     = RawKey()
                If result$ <> #Empty$
                  If Not FindString(result$, p$(1))
                    result$ = ""
                  EndIf
                EndIf
                If Not Raw
                  Delay(0)
                EndIf
              Until result$ <> ""
              result$ = result$ + ":" + Str(raw)
            EndIf
          EndIf
          If IsRuntime(p$(2))
            SetRuntimeString(p$(2), result$)
            result$ = ""
          EndIf
          
        CompilerEndIf
        
      Default
        ProcedureReturn mylookup(*udata, key$, parenthesis, isWriter) ; Cascading lookup-procedures
    EndSelect
    
    ProcedureReturn result$
  EndProcedure
  
  Procedure prime_writer(*udata, *buff, bytes)
    ;MessageRequester("Writer",string$)
    CompilerIf #USE_CONSOLE
      If *buff ; string$ <> #Null$
        If bytes ; string$ <> #Empty$
          EnableGraphicalConsole(0)
          Print(PeekS(*buff, bytes))
          EnableGraphicalConsole(1)
        EndIf
      EndIf
    CompilerEndIf
    ProcedureReturn 0
  EndProcedure
  
  Procedure$ PrimeUnescape(String$, isWriter = 0)
    If isWriter
      CompilerIf #USE_CONSOLE
        EnableGraphicalConsole(0)
      CompilerEndIf
      
    EndIf
    ProcedureReturn _UnescapeString(PrimeParse(String$, IsWriter))
  EndProcedure
  
  Procedure _Print(String$)
    CompilerIf #USE_CONSOLE
      EnableGraphicalConsole(0)
      ParseWildcards_Back(_UnescapeString(String$), 0, 0, @prime_writer(), @prime_lookup())
      EnableGraphicalConsole(0)
    CompilerEndIf
  EndProcedure
    
  Global myUsername.s
  Global myPassword.s
  Global Var1.d
  Global Var2.d
  Runtime myUsername, myPassword
  Runtime Var1, Var2
  
  Procedure Main()
    
    Protected DangerousSmiley$ = "\uD83D\uDE0A" ; <--- is this the reason?
    ;Protected DangerousSmiley$ = "" ; <--- is this the reason?
    
    
    Protected str$
    Protected uname$
    ;
    ; this caused a bug on linux with the title/header frame $(hd ...)
;     CompilerIf #PB_Compiler_OS = #PB_OS_Windows
;       uname$= "${env Username}$${fg 15}@$${fg 6}${env Computername}"
;     CompilerElse
;       uname$= "${env USER}@${rf '/etc/hostname' replace-eol}"
;     CompilerEndIf
;
    ; directly setting virtual variables (returns result from UserName() and ComputerName())
    uname$     = "$username$${fg 15}@$${fg 5}$computername$${r}"
    myUsername = ParseWildcards(ParseWildcards(uname$, @prime_lookup(), 0, 0), @prime_lookup(), 0, 0)
    ;myUsername = ParseWildcards(uname$,@prime_lookup(),0,0)
    
    str$ = "$(hd 'Hello $${fg 3}" + uname$
    str$ = str$ + "$${fg 7}!'  '-|' '+' 24)\r\n This message is \xA9 ${fg 10}free${fg 7} \U00A9.${n}" + DangerousSmiley$ + ""
    str$ = str$ + "\r\n${lset 'Physical mem free' 20 '.'}: ${inv}${rset '${rtd var1 0 .,}' 24 ' '}${inv}"
    str$ = str$ + "${n}${lset 'Total virtual'     20 '.'}: ${inv}${rset '${rtd var2 0 ,.}' 24 ' '}${inv}\r\n"
    ;str$ = str$ + "But now beware of $$ -> $$$$ $ $ $strange things can ${happen} if you don't consider ;)"
    str$ = str$ + " \r\n"
    str$ = str$ + "${lset 'Username ' 30  '.'    } : ${inv}${inln 40 '$myUsername' '_' 'myUsername'}${inv}\r\n"
    str$ = str$ + "${lset 'Password ' 30  '.' '*'} : ${inv}${inpw 40 '' '_' '*'            'myPassword'}${inv}\r\n"
    
;Define DangerousSmiley$="<no-smiley>" ; <--- is this the reason?
; Let's fill runtime variables that are accessible via the lookup procedure
    Define log$ = "./testcoloroutput.txt"
    CreateFile(0, log$, #PB_UTF8)
    If Not ReadStringFormat(0)
      WriteStringFormat(0, #PB_UTF8)
    EndIf
    WriteStringN(0, "String before parsing:")
    WriteStringN(0, str$)
    
    var1 = MemoryStatus(#PB_System_FreePhysical)
    var2 = MemoryStatus(#PB_System_TotalVirtual)
    
    CompilerIf #USE_CONSOLE
      
      OpenConsole()
      EnableGraphicalConsole(1)
      Debug MyUnescapeString(str$)
      Debug _UnescapeString(PrimeParse(str$))
      
      _Print(str$)
      
      PrintN("")
      PrintN("")
      PrintN("")
      PrintN("")
      
      PrintN("Entered Username=" + myUsername)
      PrintN("Entered Password=" + myPassword)
      
      EnableGraphicalConsole(0)
      PrintN("The same without writer, using user-defined MyUnescape() procedure returning a string (therefor no color formatting inside)")
      PrintN("")
      
      Print(PrimeUnescape(str$))
 ; Print(MyUnescapeString(str$))
      _Print("\r\nEnter ${inv}RETURN${inv} to exit\r\n")
      EnableGraphicalConsole(0)
      Input()
      
      CloseConsole()
      
    CompilerElse
      
      WriteStringN(0, "Parsed:")
      WriteStringN(0, PrimeParse(str$, 0))
      
    CompilerEndIf
    
    CloseFile(0)
    
    CompilerIf #PB_Compiler_OS = #PB_OS_Windows
      ;RunProgram(log$)
    CompilerElse
      ;RunProgram("gedit", log$, "")
    CompilerEndIf
    
  EndProcedure
  
  main()
CompilerEndIf

Re: [PB6.xx] String manipulation bug with ${escape sequences}

Posted: Sat Jun 15, 2024 9:19 am
by DarkDragon
So there are multiple ideas which come to my head, but therefore we need some information:
  • What terminal emulators do you use?
  • What shell do you use?
  • Where do you get the format string from?
  • What are the locale settings?

Re: [PB6.xx] String manipulation bug with ${escape sequences}

Posted: Sat Jun 15, 2024 11:08 am
by benubi
To answer that I have to learn a few things about Linux first. I needed a bit of exercise anyway :lol:

I only use the pre-installed programs so far.

Shell+Terminal emulators:
- bash+"LXTerminal 0.4.0 Terminal emulator for LXDE project" (Raspberry 4 arm-64)

- bash+"GNOME Terminal 3.46.8 for GNOME 43" (Debian x64)


Format string: I am not sure what you exactly mean. For now this is a "standalone" include that isn't finished. I hard code the ${xyz} escape sequences in a string or write them simply in a text file.

Locales:
- en_US.UTF-8 (raspberry pi arm64)
- de_DE.UTF-8 (Debian x64 VM)


There is still a part that runs on the RPi machine; for example the 2 messageboxes are displayed, and the second half of the demo text is displayed. But both linux machines they have trouble rendering the "header" formater, which puts a string into a box like that

Code: Select all

**********
* demo 1 *
**********

+-------+
| Demo2 |
+-------+

etc. are only correctly rendered on windows
To be able to put the title in the box, even with colors, I have to filter out all the $ escape sequences; this happens by Parsing recursively the string parameter containing the title - up to 4 times.

Either the recursion causes the strings to be corrupted (kernel switch), or they come in corrupted (compiler switch); then the frame around the title seems to contain letters that appear at other places in the "parent string" from which the command & arguments were extracted "like CLI parameters".(I never heard or don't know anything about those kind switches, or if they exist - it's just a speculation)

In the Main() procedure the String "str$" contains the escape sequences/formatters that make up most of the demo code (except the 2 messagebox calls part).

I wonder if it's on the c compiler/backend side or maybe a secret kernel switch that happens, because it happens more "around" Print() and graphical console commands.

The ParseWildcards() that returns a string has better integrity than the ParseWildcards_Back() that uses a writer function instead of returning a string; the writer uses the console commands to render the text in colors etc.

I will have to make an other demo/test code that has 0 console commands.

Re: [PB6.xx] String manipulation bug with ${escape sequences}

Posted: Sat Jun 15, 2024 12:38 pm
by Fred
I don't think it's a PB bug actually.

Re: [PB6.xx] String manipulation bug with ${escape sequences}

Posted: Mon Jun 17, 2024 1:41 pm
by benubi
I updated the code in Post #3

The bug seems not to be related to Console/Terminal, this was a false hint. Sorry. I would not exclude it completely, that there may be some magic "bash injection" happening but I don't know even what I am imagining ;)

Now the header formatting works a little better concerning the frame drawing, "because" the username is now constructed differently.

I modified the code and in main() and (look-up procedures) you can see that before I read the /etc/hostname file on Linux, now the lookup function uses directly UserName() and ComputerName() when the variables $username and $computername are looked up.

But this happened when I changed it: the username would pass through, but the computername sometimes not!

When I changed

Code: Select all

ProcedureReturn ComputerName()
to

Code: Select all

result$=ComputerName()
; (... select/endselect)
ProcedureReturn result$
results improved but now on Linux x64 the computer name is empty after the @ in the user name; it seems to be tabulator and not spaces since I can't mouse select parts of it, or maybe it's a console color trick but it looks very void (there are no tabs in the escape sequences/string).

IMO it has to do with recursion and returning strings from "far away" - when a prototype.s returns a string from an other prototype.s etc. in a chain "recursively"; it remembers me the problem with strings and DLL's. I had similar bugs I could not clearly define when using interfaces. Suddenly strings won't be returned or are broken. But these problems always arise when the code already has reached a greater length and complexity.
:cry:

Re: [PB6.xx] String manipulation bug with ${escape sequences}

Posted: Mon Jun 17, 2024 3:51 pm
by Fred
Well, without a reproducing snippet, it's impossible to fix as it could be a side effect from another error.

Re: [PB6.xx] String manipulation bug with ${escape sequences}

Posted: Tue Jun 18, 2024 12:35 pm
by benubi
I updated the code in post #3 again; little change the "header/title" function now takes a 4th parameter for indentation.

The code above produces faulty output on Raspberry (right) and the expected output on the other systems. My Debian VM terminal display yellow color (6) in black which is why the computer name looked void but I changed to color 5 in the formatter string, as the Terminal refuses to display yellow (thanks to HTML-conversion feature in the menu I found out the computer name is not void but "only invisible").
https://ibb.co/T129zpx

The faulty part is processed with ParseWildcard_Back(), written chunk/token -wise using print().

The "good" part is processed with ParseWildcard(), which returns the parsed content a string (no print/console operations involved).

I'll try this next:
- try to make/find the bug in a much shorter example
- draw text on a bitmap instead of printing in console
- change the OS for the raspberry (needs to be done to use new 6.11+ versions).

I think it's a raspberry installation related bug or side effect, either on the kernel or in the static libraries the compiler may use. I fear that I could have stolen much time for a bug I "invented" but it makes no sense that it works "flawlessly" on all systems except the raspberry. The only difference is the byte ordering on RPi is big endian but this shouldn't be of any importance for string manipulation.

Re: String manipulation with ${escape sequences}

Posted: Tue Jan 14, 2025 7:54 pm
by benubi
UPDATE

Finally it works. I made an OS upgrade on the RPi, all by myself using the text editor and the console ;)

Now I can start PB6.11 and later versions, on the later version the string problem is resolved.

But when I use the old PB6.00 version, new compiled programs still have that bug, and the old compiled programs also display the bug after upgrade to the new OS.

From my POV this weird behavior happens during compilation or linking, and while IDK what the reason is I could imagine a third party library locally installed to be the culprit (or configuration), since the problem could not be reproduced on other machines but it still persists after my OS upgrade (on the Raspberry Pi 400).

It remains mysterious but for me I consider the problem "fixed" :P
All times are UTC+01:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited