Parsing Tokens

[Killswitch]

I am currently in the midst of writing a programming language in PureBasic it's self, and I'm having a little bit of trouble parsing commands which have nested commands within them.

I'm trying to convert:

Command1( Command2(1,2,3 )+1 , Aurg )

Into:

1 2 3 Command2() +1 ¬ Aurg Command1() ¬

Basically I'm trying to put the most nested command first, so I can work out its result before the others. The format is:

<Parameters> Command() Maths BREAK

Does anyone have any idead of how to do this successfully, it'd really help me a lot.

Here is my best working example:

Code: Select all

 Procedure.s ParseToken(Recursive,Token.s)
      
      Stack.s
      Scanned.s
      Temp.s
      
      If Recursive=0 And CountString(Token,"(")>1
        Stack+Mid(Token,1,FindString(Token,"(",1)-1)+"()"+Chr(190)+" "
        Token=Mid(Token,FindString(Token,"(",1)+1,Len(Token))
      EndIf
      
      Debug Token
      
      While Token
        Scanned=Left(Token,1)        
        Token=Mid(Token,2,Len(Token))
        
        If Scanned="("
          If Temp
            Stack=Temp+"()"+Chr(190)+" "+Stack
          EndIf
          If Recursive=0
            Temp=ParseToken(1,Mid(Token,1,FindString(Token,")",1)))
            Stack=Temp+Stack
          ElseIf Recursive=1
            Temp=ParseToken(1,Mid(Token,1,FindString(Token,")",1)))
            Stack=Temp+Stack
            ;ProcedureReturn Stack
          EndIf
          Temp=""
          Token=Mid(Token,FindString(Token,")",1)+1,Len(Token))
        ElseIf Scanned=","
          If Recursive
            If Temp
              Stack=Temp+" "+Stack
              Temp=""
            EndIf
          Else
            If Temp
              Stack+Temp+" "
              Temp=""
            EndIf
          EndIf
        ElseIf Scanned=")"
          If Token
            Debug 1
          EndIf
          If Recursive 
            If Temp
              Stack=Temp+" "+Stack
            EndIf
          Else
            If Temp
              Stack+Temp+" "
            EndIf
          EndIf
          ;ProcedureReturn Stack
        ElseIf Scanned<>"()"
          Temp+Scanned
        EndIf
          
        
      Wend
      
      ProcedureReturn Stack
      
    EndProcedure

[theNerd]

Here is a tool for creator language parsers. Perhaps you could convert the C engine to PureBasic?

http://www.devincook.com/goldparser/

[Xombie]

What about something like...

Code: Select all

Procedure.s ParseCommands(inString.s)
   ;
   WorkingString.s = inString
   ;
   HoldLength.l = Len(inString)
   ;
   iLoop.l
   ;
   HoldString.s
   ;
   PositionClose.l : PositionOpen.l : PositionCurrent.l
   ;
   Character.b
   ;
   CurrentCommand.s
   ;
   *MemPosition.l = @WorkingString + HoldLength
   ;
   *MemCommandName.l
   ;
   HoldCommandName.s : HoldArguments.s
   ;
   ReturnString.s
   ;
   IsFinished.b
   ;
   Repeat
      ;
      Character = PeekB(*MemPosition)
      ;
      If Character = 40
         ; '(' Character - The start of a procedure.
         PositionOpen = *MemPosition - @WorkingString
         ;
         *MemCommandName = *MemPosition
         ;
         Repeat
            *MemCommandName - 1
            If PeekB(*MemCommandName) = 40
               *MemCommandName + 1
               Break
            EndIf
            If *MemCommandName = @WorkingString : IsFinished = #True : EndIf
         Until *MemCommandName = @WorkingString
         ; Repeat until we locate the command name.
         HoldString = Mid(WorkingString, PositionOpen + 2, PositionClose - PositionOpen - 1)
         ;
         ReturnString = ReturnString + ReplaceString(HoldString, ",", " ")
         ; Add the arguments.
         ReturnString = ReturnString + Chr(30)
         ;
         ReturnString = ReturnString + Mid(WorkingString, *MemCommandName - @WorkingString + 1, *MemPosition - *MemCommandName)
         ; Copy our command name.
         ReturnString = ReturnString + "()"
         ;
         ReturnString = ReturnString + Chr(31)
         ; Add a break for every command.
         WorkingString = Mid(WorkingString, 1, *MemCommandName - @WorkingString) + Mid(WorkingString, PositionClose + 2, HoldLength - PositionClose)
         ; Now slice out the command we've just worked on so it doesn't get touched next.
         HoldLength = Len(WorkingString)
         ;
         *MemPosition = @WorkingString + HoldLength
         ;
      ElseIf Character = 41
         ; )
         PositionClose = *MemPosition - @WorkingString
         ;
      EndIf
      ;
      *MemPosition - 1
      ;
   Until *MemPosition = @WorkingString Or IsFinished = #True
   ;
   ProcedureReturn ReturnString
   ;
EndProcedure

a.s = " Command1( Command2(1,2,3 )+1 , Aurg )"
Debug ParseCommands(a)

The problem with your example is you aren't consistent. Your break is in the wrong place. You'd almost want to do something like...

Argument <BREAK ASCII CHAR 30> CommandName <BREAK ASCII CHAR 31>

And repeat, wouldn't you? If a command is an argument of another command you'd want to solve the first command and then it is simplified into a simple argument. I could be mistaking what you're trying to do, though.

Anyhow, hope something in there helps you. Good luck.

[utopiomania]

Is the language you work on something like an interpreted script language/basic variant or something more advanced ??

[dmoc]

IIRC the way to do this is parse the expression into a tree and then parse the tree