Read JSON File

[IdeasVacuum]

Struggling with JSON, never used the PB JSON lib before. I am going to work with large-ish JSON files produced by others. No two files are the same, so the structure or order of the data is an unknown. They each define a window/form. Components (Gadgets) on the form can be in containers, which can themselves be in containers, but the sample file linked does not have that complexity.

Sample.txt

What I need to do is read the file, extract all the Key-Value pairs, modify some of them, write a new file. I'm stuck on the extraction bit!

Using the code examples in the PB Help, if I cut-out a portion of the file between and including {} brackets (an Object), I can produce a list of Key-Value pairs from that snippet. If however I try to process the whole file string, only JSON Objects are found and if I try to extract Key-Value pairs from the Objects, my code fails:

Code: Select all

Enumeration
#FileIO
#Jason
EndEnumeration

Global sFile.s = "C:\Sample.txt"

Structure JSON_objects
sObjKey.s
iObjSize.i
EndStructure

Global igJval.i

Global NewList gJsonObjectsList.JSON_objects()
Global NewList gJsonSubObjectsList.JSON_objects()


Procedure.s PfReadFile()
;#----------------------
Protected sSkip.s, sJason.s

              If ReadFile(#FileIO, sFile)

                        sSkip = ReadString(#FileIO, #PB_Ascii)
                        sSkip = ReadString(#FileIO, #PB_Ascii)
                       sJason = ReadString(#FileIO, #PB_Ascii)

                       CloseFile(#FileIO)

                       ProcedureReturn(sJason)
              Else
                       MessageRequester("Fail","Could not open file")
                       ProcedureReturn "FAIL"
              EndIf
EndProcedure

Procedure PfParseObjs()
;#----------------------
Protected      iItem.i
Protected Dim sArray.s(0)

              ForEach gJsonObjectsList()

                        For iItem = 0 To gJsonObjectsList()\iObjSize

                             ParseJSON(#Jason, gJsonObjectsList()\sObjKey)

                                                   igJval = JSONValue(#Jason)
                             If ExamineJSONMembers(igJval)

                                   Select JSONType(igJval)

                                          Case    #PB_JSON_Null: Debug "Null"
                                          Case  #PB_JSON_String: Debug JSONMemberKey(igJval) + ": " + GetJSONString(GetJSONMember(igJval, JSONMemberKey(igJval)))
                                          Case  #PB_JSON_Number: Debug JSONMemberKey(igJval) + ": " + StrD(GetJSONDouble(GetJSONMember(igJval, JSONMemberKey(igJval))),4)
                                          Case #PB_JSON_Boolean: Debug JSONMemberKey(igJval) + ": " + Str(GetJSONBoolean(GetJSONMember(igJval, JSONMemberKey(igJval))))
                                          Case   #PB_JSON_Array
                                                                 Debug JSONMemberKey(igJval) + ": "
                                                                 ExtractJSONArray(GetJSONMember(igJval, JSONMemberKey(igJval)), sArray())

                                                                 For iItem = 0 To ArraySize(sArray())

                                                                          Debug sArray(iItem)
                                                                 Next
                                          Case #PB_JSON_Object

                                                                 AddElement(gJsonSubObjectsList())
                                                                 gJsonSubObjectsList()\sObjKey  = JSONMemberKey(igJval)
                                                                 Debug gJsonSubObjectsList()\sObjKey
                                                                 gJsonSubObjectsList()\iObjSize = JSONObjectSize(igJval)
                                   EndSelect
                             EndIf
                        Next
              Next
EndProcedure

Procedure PfJason()
;#-----------------
Protected iItem.i
Protected Dim sArray.s(0)
Protected  sJason.s = PfReadFile()

              If Not (sJason = "FAIL")

                       ParseJSON(#Jason, sJason)

                       igJval = JSONValue(#Jason)

                       If ExamineJSONMembers(igJval)

                                While NextJSONMember(igJval)

                                        Select JSONType(igJval)

                                          Case    #PB_JSON_Null: Debug "Null"
                                          Case  #PB_JSON_String: Debug JSONMemberKey(igJval) + ": " + GetJSONString(GetJSONMember(igJval, JSONMemberKey(igJval)))
                                          Case  #PB_JSON_Number: Debug JSONMemberKey(igJval) + ": " + StrD(GetJSONDouble(GetJSONMember(igJval, JSONMemberKey(igJval))),4)
                                          Case #PB_JSON_Boolean: Debug JSONMemberKey(igJval) + ": " + Str(GetJSONBoolean(GetJSONMember(igJval, JSONMemberKey(igJval))))
                                          Case   #PB_JSON_Array
                                                                 Debug JSONMemberKey(igJval) + ": "
                                                                 ExtractJSONArray(GetJSONMember(igJval, JSONMemberKey(igJval)), sArray())

                                                                 For iItem = 0 To ArraySize(sArray())

                                                                          Debug sArray(iItem)
                                                                 Next
                                         Case #PB_JSON_Object

                                                                 AddElement(gJsonObjectsList())
                                                                 gJsonObjectsList()\sObjKey  = JSONMemberKey(igJval)
                                                                 Debug gJsonObjectsList()\sObjKey
                                                                 gJsonObjectsList()\iObjSize = JSONObjectSize(igJval)
                                        EndSelect
                                Wend
                       EndIf
              EndIf

              Debug "==============================="

              If(ListSize(gJsonObjectsList()) > 0) : PfParseObjs() : EndIf
EndProcedure

PfJason()
End

[/size]

It seems to me that there should be a better way to approach this task anyway. I'm wondering though if the JSON format of the file fully complies with the standard that the PB Lib expects.......

[normeus]

I know I used PB with JSON once but I cannot find my program.
Have a look at this post by Freak which might help you

http://www.purebasic.fr/english/viewtop ... 20#p467020


Norm.

[IdeasVacuum]

Thanks Norm - that has got potential

[IdeasVacuum]

Well, looking at other posts and my early tests, there is an inherent JSON format PITA in that for some purposes it is essential that the order of the data (Key-Value pairs) remains the same but the Lib does not deliver that. I can see it often does not matter, for example defining the display of the info can be structured as you wish, but if writing a replacement file, It's probably best that the order of data matches the original file to avoid potentially undetected faults/data exchange failure.

Even though the complexity increases as the files I need to process get bigger, there is a human-readable data order in there.

So, I'm going to persevere with the JSON lib to understand it better, but looks like the ultimate solution in my case will have to use the String Lib.....but I'm running out of hair to turn gray.

[davido]

@IdeasVacuum,

You might find this of interest:
http://www.purebasic.fr/english/viewtop ... 46#p511346

[kenmo]

Well, looking at other posts and my early tests, there is an inherent JSON format PITA in that for some purposes it is essential that the order of the data (Key-Value pairs) remains the same but the Lib does not deliver that.

The PB JSON library is very nice, and it's easy to extend it with custom procedures, but it's a shame that it doesn't maintain the order of object members
This makes files less readable, harder to diff, and incompatible with some applications, as discussed here and other threads!

I guess it uses unordered Maps internally instead of ordered Lists for storage.

I wrote a JSON module in 2014 that DOES maintain order, right before PB added its JSON library.
Maybe it's useful for you (but it needs modifications to compile in PB 5.6)
https://raw.githubusercontent.com/kenmo ... r/JSON.pbi


EDIT: If you're interested, I edited my old JSON.pbi into OJSON.pbi which no longer conflicts names with PB's JSON library. This will preserve Object order:
https://raw.githubusercontent.com/kenmo ... /OJSON.pbi

[IdeasVacuum]

Hi Guys

I have learnt a lot from both Little John and Kenmo's work and I believe that PB's JSON Lib should be upgraded to follow their lead. As-is, it's more than a bit annoying!

However, the larger files I'm processing are difficult to drill-down (and seemingly not 100% JSON compliant), especially where there are components within components within components. So, I'm putting together bespoke code dedicated to these files, which vary greatly in content.

[kenmo]

Yeah, that's another thing I've seen. PB's library requires strict standard JSON, but lots of sources of JSON files break the syntax rules and add non-standard content... I would say this is not a PB problem. In fact I have written code to "normalize" JSON files into the strict format.

[IdeasVacuum]

Indeed, but PB is a little naughty too:

Any '*' or '$' characters are stripped from the structure member names before comparing them to the JSON object members. So a member key must not include these characters to be properly matched to a structure member.

It's obvious why PB doesn't want those symbols but it's a good practice not to butcher data to suit your tools - if the JSON files are produced by a 3rd party, you likely have no influence over the use of symbols but their inclusion could be critical to the meaning of the data.....