Generate PB Structures for JSON data

[freak]

Inspired by a recent question (http://www.purebasic.fr/english/viewtop ... 13&t=62501), I wrote a little code generator that automates the process of creating PB structures for JSON data.

You just need to enter some example JSON and it will generate structure(s) which can be used with the ExtractJSONStructure() command to easily access the data. The JSON data must be an object (with {}). If your data is represented as an array (with []), just generate the structure(s) for the first item and then use ExtractJSONArray() or ExtractJSONList() to get the data later.

Make sure to use JSON data with all values filled for the code generation, because the generator cannot determine the type to use for null values or empty arrays.

Code: Select all

#NewLine = Chr(13) + Chr(10)

Structure JsonObject
  Name$
  Value.i
EndStructure

Global NewList Objects.JsonObject()
Global GeneratedStructures

Declare.s GenerateStructure(JsonValue)

Runtime Enumeration
  #JsonGadget
  #CodeGadget
  #PrefixGadget
  #NumberTypeGadget
  #StringTypeGadget
  #BooleanTypeGadget
  #NullTypeGadget
  #UseListsGadget
EndEnumeration

; Compare two JSON values recursively to see if they have the same structure
;
Procedure CompareJson(Value1, Value2)

  If JSONType(Value1) <> JSONType(Value2)
    ProcedureReturn #False
  EndIf
  
  If JSONType(Value1) = #PB_JSON_Array
    If JSONArraySize(Value1) = 0 And JSONArraySize(Value2) = 0
      ProcedureReturn #True
    ElseIf JSONArraySize(Value1) > 0 And JSONArraySize(Value2) > 0
      ProcedureReturn CompareJson(GetJSONElement(Value1, 0), GetJSONElement(Value2, 0))
    Else
      ProcedureReturn #False
    EndIf
  
  ElseIf JSONType(Value1) = #PB_JSON_Object
     If JSONObjectSize(Value1) <> JSONObjectSize(Value2)
       ProcedureReturn #False
     EndIf
     
     If ExamineJSONMembers(Value1)
       While NextJSONMember(Value1)
         OtherValue = GetJSONMember(Value2, JSONMemberKey(Value1))
         If OtherValue = 0 Or CompareJson(JSONMemberValue(Value1), OtherValue) = #False
           ProcedureReturn #False
         EndIf
       Wend
     EndIf
  
  EndIf

  ProcedureReturn #True
EndProcedure

; Returns true if a JSON value of type Object contains names that can be converted to structure members
;
Procedure ValidStructure(JsonValue)
  Protected NewList Seen.s()

  If ExamineJSONMembers(JsonValue)
    While NextJSONMember(JsonValue)
      Name$ = LCase(JSONMemberKey(JsonValue))
      
      ; check for empty name
      If Name$ = ""
        ProcedureReturn #False
      EndIf
      
      ; check for ambiguous names within the structure (only different by case)
      ForEach Seen()
        If Seen() = Name$
          ProcedureReturn #False
        EndIf
      Next Seen()      
      AddElement(Seen())
      Seen() = Name$
      
      ; check for invalid start char
      If FindString("abcdefghijklmnopqrstuvwxyz_", Left(Name$, 1)) = 0
        ProcedureReturn #False
      EndIf
      
      ; check for other invalid chars
      For i = 1 To Len(Name$)
        If FindString("abcdefghijklmnopqrstuvwxyz_1234567890", Mid(Name$, i, 1)) = 0
          ProcedureReturn #False
        EndIf
      Next i
      
    Wend
  EndIf  

  ProcedureReturn #True
EndProcedure

; Get the PB type suffix for a JSON value
;
Procedure.s GetTypeSuffix(JsonValue)
  Select JSONType(JsonValue)
  
    Case #PB_JSON_Null 
      ProcedureReturn GetGadgetText(#NullTypeGadget)
    
    Case #PB_JSON_String 
      ProcedureReturn GetGadgetText(#StringTypeGadget)
    
    Case #PB_JSON_Number 
      ProcedureReturn GetGadgetText(#NumberTypeGadget)
    
    Case #PB_JSON_Boolean 
      ProcedureReturn GetGadgetText(#BooleanTypeGadget)
    
    Case #PB_JSON_Array 
      If JSONArraySize(JsonValue) = 0
        ProcedureReturn GetGadgetText(#NullTypeGadget) ; Type unknown because the array is empty
      Else
        ProcedureReturn GetTypeSuffix(GetJSONElement(JsonValue, 0))
      EndIf
    
    Case #PB_JSON_Object     
      ; See if the structure already exists
      ForEach Objects()
        If CompareJson(JsonValue, Objects()\Value)
          ProcedureReturn "." + Objects()\Name$
        EndIf
      Next Objects()    
    
      ; Generate a new structure
      ProcedureReturn "." + GenerateStructure(JsonValue)
          
  EndSelect
EndProcedure

Procedure.s GenerateStructure(JsonValue)
  Protected NewList Members.s()

  ; Get structure name
  If GeneratedStructures = 0
    StructureName$ = GetGadgetText(#PrefixGadget)
  Else
    StructureName$ = GetGadgetText(#PrefixGadget) + "_" + Str(GeneratedStructures)
  EndIf
  GeneratedStructures + 1
  
  ; Get the members, generate any sub-structures
  If ExamineJSONMembers(JsonValue)
    While NextJSONMember(JsonValue)
      ItemName$ = JSONMemberKey(JsonValue)
      ItemValue = JSONMemberValue(JsonValue)
      
      AddElement(Members())
      If JSONType(ItemValue) = #PB_JSON_Object
        If ValidStructure(ItemValue) = #False And ExamineJSONMembers(ItemValue) And NextJSONMember(ItemValue)
          Members() = "Map " + ItemName$ + GetTypeSuffix(JSONMemberValue(ItemValue)) + "()"          
        Else
          Members() = ItemName$ + GetTypeSuffix(ItemValue)
        EndIf
      ElseIf JSONType(ItemValue) = #PB_JSON_Array      
        If GetGadgetState(#UseListsGadget)
          Members() = "List " + ItemName$ + GetTypeSuffix(ItemValue) + "()"
        Else
          Members() = "Array " + ItemName$ + GetTypeSuffix(ItemValue) + "(0)"
        EndIf
      Else
        Members() = ItemName$ + GetTypeSuffix(ItemValue)
      EndIf
    Wend
  EndIf
  
  ; Now output the structure (any sub-structures were already added to the output)
  AddGadgetItem(#CodeGadget, -1, "Structure " + StructureName$)
  ForEach Members()
    AddGadgetItem(#CodeGadget, -1, "  " + Members())
  Next Members()
  AddGadgetItem(#CodeGadget, -1, "EndStructure")
  AddGadgetItem(#CodeGadget, -1, "")
  
  ; Register the structure for re-use
  AddElement(Objects())
  Objects()\Name$ = StructureName$
  Objects()\Value = JsonValue
  
  ProcedureReturn StructureName$
EndProcedure

Runtime Procedure Generator()
  GeneratedStructures = 0
  ClearList(Objects()) 
  ClearGadgetItems(#CodeGadget) 
  
  If ParseJSON(0, GetGadgetText(#JsonGadget))
    If JSONType(JSONValue(0)) = #PB_JSON_Object    
      GenerateStructure(JSONValue(0))      
    Else
      Code$ = "; Main JSON Element is not of type #PB_JSON_Object"
      SetGadgetText(#CodeGadget, Code$)
    EndIf  
  Else
    Code$ = "; " + JSONErrorMessage() + #NewLine + 
            "; Line " + JSONErrorLine() + " Column " + JSONErrorPosition()
    SetGadgetText(#CodeGadget, Code$)
  EndIf
EndProcedure

Dialog$ = "<window name='generator' text='JSON Structure Generator' flags='#PB_Window_SystemMenu | #PB_Window_ScreenCentered | #PB_Window_SizeGadget | #PB_Window_MinimizeGadget | #PB_Window_MaximizeGadget'>" + 
          "  <vbox>" +
          "    <splitter>" +
          "      <frame text='JSON Data:'>" +
          "        <editor id='#JsonGadget' width='600' height='160'/>" +
          "      </frame>" +
          "      <frame text='PB Code:'>" +
          "        <editor id='#CodeGadget' width='600' height='160' flags='#PB_Editor_ReadOnly'/>" +    
          "      </frame>" +    
          "    </splitter>" +
          "    <frame text='Generator'>" +
          "      <hbox expand='item:2'>" +
          "        <gridbox columns='5' colexpand='no'>" +
          "          <text text='PB Type for Numbers: '/>" +
          "          <string id='#NumberTypeGadget' text='.i' width='50'/>" +
          "          <empty width='30'/>" +
          "          <text text='Structure Prefix: '/>" +
          "          <string id='#PrefixGadget' text='Json' width='100'/>" +
          "          <text text='PB Type for Strings: '/>" +
          "          <string id='#StringTypeGadget' text='$'/>" + 
          "          <empty/>" +
          "          <checkbox id='#UseListsGadget' text='Use Lists instead of Arrays' colspan='2'/>" +
          "          <text text='PB Type for Booleans: '/>" +
          "          <string id='#BooleanTypeGadget' text='.i'/>" + 
          "          <empty colspan='3'/>" +
          "          <text text='PB Type for Nulls: '/>" +
          "          <string id='#NullTypeGadget' text='$'/>" + 
          "          <empty colspan='3'/>" +
          "        </gridbox>" +
          "        <singlebox expand='no' margin='0' align='top,right'>" +
          "          <button onevent='Generator()' text='Generate'/>" +
          "        </singlebox>" +
          "      </hbox>" +
          "    </frame>" +
          "  </vbox>" +
          "</window>"

If ParseXML(0, Dialog$) And XMLStatus(0) = #PB_XML_Success And CreateDialog(0) And OpenXMLDialog(0, 0, "generator")
  While WaitWindowEvent() <> #PB_Event_CloseWindow: Wend
  End
Else
  Debug XMLError(0)
EndIf

[Kiffi]

very useful, thanks a lot!

Greetings ... Peter

[infratec]

This avoids some new gray hairs on my head

But...

After a few tests, I noticed that it fails when a map or a list is requierd in the structure.


Like here:
http://www.purebasic.fr/english/viewtop ... 13&t=61754

Bernd

[freak]

Well, since a structure and a map are the same in JSON the generator cannot know what you want. So you will need to do some editing by hand if you want something different.

[Little John]

very useful, thanks a lot!

+ 1

Well, since a structure and a map are the same in JSON the generator cannot know what you want.

Yes, in many cases the generator cannot know what we want.

However, I think sometimes it actually could know it, so here is a small suggestion for improvement.
When entering the following valid JSON data

Code: Select all

{
  "categories": {
      "1": 527,
      "2": 412,
      "3": 739
    }
}

into your generator, it produces this result:

Code: Select all

Structure Json_1
  1.i
  2.i
  3.i
EndStructure

Structure Json
  categories.Json_1
EndStructure

The generated structure is wrong, since numbers are not valid names for PB structure fields.
So in cases like this, the only valid result is a map:

Code: Select all

Structure Json
   Map categories.i()
EndStructure

As a side note (not addressed to freak):
The result can't be an array or a list, since in JSON both look like this:

Code: Select all

{
  "categories": [
      527,
      412,
      739
    ]
}

[Danilo]

When entering the following valid JSON data

Code: Select all

{
  "categories": {
      "1": 527,
      "2": 412,
      "3": 739
    }
}

into your generator, it produces this result:

Code: Select all

Structure Json_1
  1.i
  2.i
  3.i
EndStructure

Structure Json
  categories.Json_1
EndStructure

The generated structure is wrong, since numbers are not valid names for PB structure fields.
So in cases like this, the only valid result is a map:

Code: Select all

Structure Json
   Map categories.i()
EndStructure

What's the problem with converting numbers like 1,2,3 into variables _1, _2, _3, or var_1, var_2, var_3, etc.?

[Little John]

What's the problem with converting numbers like 1,2,3 into variables _1, _2, _3, or var_1, var_2, var_3, etc.?

Did someone say that it would be a problem?
Fact is, that freak's generator currently does not do such a conversion, and thus generates invalid structures in cases like the one I mentioned.
If renaming the structure fields is an option, then it could generate valid structures in these cases, of course.

[freak]

I changed the code to generate a map if any of the object members cannot be used as a valid structure name.

What's the problem with converting numbers like 1,2,3 into variables _1, _2, _3, or var_1, var_2, var_3, etc.?

Since many languages have similar rules regarding allowed names in code, it is unlikely that an API returning an object containing "1" as a member is intended to be converted to a structure. So a map is probably the right guess then.

But as I said above, the result can always be edited to fit individual needs.

[Little John]

Thanks again, freak!

[davido]

Very nice.
Thank you, very much.

[said]

Thank you freak Very nice

Any code snippet/tip coming from the team is always most welcome
Nice catch Little John

[minimy]

Thanks, very nice!
+1

[c4s]

Thanks for this tool. I just needed it and it works like a charm!

I'm no XML expert but noticed that we also have ExtractXMLStructure(). Does it mean that this code could easily be extended to support XML data as well?