It is currently Sat Feb 24, 2018 9:11 pm

All times are UTC + 1 hour




Post new topic Reply to topic  [ 9 posts ] 
Author Message
 Post subject: Save JSON data with object members well-arranged (Module)
PostPosted: Tue Sep 05, 2017 7:49 pm 
Offline
Addict
Addict
User avatar

Joined: Thu Jun 07, 2007 3:25 pm
Posts: 3145
Location: Berlin, Germany
Hi,

one key point of the JSON data format is, that the data are both machine readable and human readable.
However, there is a serious problem:
A JSON object by definition is an unordered collection of name/value pairs. And PureBasic's built-in function SaveJSON() saves the members of objects in an unpredictable way, it's not possible to control their order. This can make the generated JSON files hard to read for humans. I, for instance, not seldom want to compare 2 JSON files visually, using a program such as the built-in file compare tool. This only makes sense, if the "fields" in both files are in the same order.

This "JSave" module solves the problem. It works on all platforms supported by PureBasic (many thanks to davido for testing on Mac!). It can be easily used in a simple or in a more advanced way.

In the simplest case, just write
Code:
JSave::Save(jsonId, outFile$)
to save all member names of all objects sorted ascending.

Or use
Code:
JSave::Save(jsonId, "")
for displaying the JSON data in the Debug window.

The code
Code:
JSave::Save(jsonId, outFile$, #PB_Sort_Descending)
will save all member names of all objects sorted descending.

Using InitObject() or InitObjectStr(), you can for each object define the names of known members. This defines the order in which these members will be saved (see demo code of the module). All unknown members will still be sorted according to the sorting mode applied to that object. You can even choose to ignore unknown members.

Any constructive feedback will be appreciated.
Enjoy!

// Changes on 2018-02-15:
  • Fixed a bug
  • Demo code slightly enhanced

Code:
; -- Save (or show with Debug) JSON data pretty-printed, with object
;    members individually arranged or sorted according to their names
;    (array elements are not affected).
;    ==> can replace the built-in function SaveJSON()
;
; tested with PB 5.60 x86 and x64 on Windows 10,
;             PB 5.45 LTS x86 on Linux Mint 18.2 Cinnamon,
;             PB 5.50 on Mac OS X (many thanks to davido);
; by Little John, most recently changed 2018-02-15
; <http://www.purebasic.fr/english/viewtopic.php?f=12&t=69100>

CompilerIf #PB_Compiler_Version < 540
   CompilerError "PureBasic version 5.40 or newer required"
CompilerEndIf


DeclareModule JSave
   Declare.i InitObject (objectName$, List memberName$(), sortUnknown.i=#PB_Sort_Ascending)
   Declare.i InitObjectStr (objectName$, memberList$, sortUnknown.i=#PB_Sort_Ascending)
   Declare   Clear()
   
   Declare.i Save (json.i, file$, sortUnknown.i=#PB_Sort_Ascending)
EndDeclareModule


Module JSave
   EnableExplicit
   
   Structure Object
      Map KnownMember.i()
      List *known.String()
      SortUnknownMembers.i
   EndStructure
   
   NewMap s_Object.Object()
   Define s_SortUnknownObjects.i, s_Ofn.i, s_CurMemberKey$
   
   
   Procedure.i InitObject (objectName$, List memberName$(), sortUnknown.i=#PB_Sort_Ascending)
      ; -- For each wanted object, define the order in which its members should be saved to a JSON file
      ;    (optional function).
      ; in : objectName$  : name of regarding object
      ;                     ("" refers to members at the basic level)
      ;      memberName$(): list of names of known members for this object;
      ;                     This list defines the order of the members.
      ;      sortUnknown  : object specific setting for sorting unknown members
      ;                     (any PureBasic sort options for strings,
      ;                      or -1 for ignoring unknown members)
      ; out: return value : 1 on success,
      ;                     0 on error
      Shared s_Object()
     
      If FindMapElement(s_Object(), objectName$)
         ProcedureReturn 0        ; error
      EndIf
     
      AddMapElement(s_Object(), objectName$, #PB_Map_NoElementCheck)
     
      With s_Object()
         ForEach memberName$()
            AddElement(\known())                                                       ; Add an element to the list of the new object,
            \known() = AddMapElement(\KnownMember(), memberName$()) - SizeOf(Integer)  ; and store the pointer to the new mapkey there.
         Next
         \SortUnknownMembers = sortUnknown
      EndWith
     
      ProcedureReturn 1           ; success
   EndProcedure
   
   
   Procedure.i InitObjectStr (objectName$, memberList$, sortUnknown.i=#PB_Sort_Ascending)
      ; -- Wrapper for function InitObject(), for convenience
      ;    (optional function).
      ; in : objectName$ : name of regarding object
      ;                    ("" refers to members at the basic level)
      ;      memberList$ : list of names of known members for this object, separated by ',' ;
      ;                    This list defines the order of the members.
      ;      sortUnknown : object specific setting for sorting unknown members
      ;                    (any PureBasic sort options for strings,
      ;                     or -1 for ignoring unknown members)
      ; out: return value: 1 on success,
      ;                    0 on error
      Protected numFields.i, i.i
      Protected NewList memberName$()
     
      numFields = CountString(memberList$, ",") + 1
      For i = 1 To numFields
         AddElement(memberName$())
         memberName$() = Trim(StringField(memberList$, i, ","))
      Next
     
      ProcedureReturn InitObject(objectName$, memberName$(), sortUnknown)
   EndProcedure
   
   
   Procedure Clear()
      ; -- Clear the map of known objects
      Shared s_Object()
     
      ClearMap(s_Object())
   EndProcedure
   
   
   Macro _WriteLine (_key_, _right_)
      If _key_ = ""
         line$ = pre$ + _right_
      Else
         line$ = pre$ + LSet(#DQUOTE$ + _key_ + #DQUOTE$, fill) + ": " + _right_
      EndIf
     
      If s_Ofn
         WriteStringN(s_Ofn, line$)
      Else
         Debug line$
      EndIf
   EndMacro
   
   Procedure _TraverseJSON (v.i, level.i, fill.i, key$="", comma$="")
      Shared s_Object(), s_SortUnknownObjects, s_Ofn, s_CurMemberKey$
      Protected NewList unknownMember$()
      Protected.i i, last, pad, validMembers, knownObject
      Protected tmp$, line$, pre$ = Space(3 * level)
     
      Select JSONType(v)
         Case #PB_JSON_Object
            If JSONObjectSize(v) = 0
               _WriteLine(key$, "{}" + comma$)
               
            Else
               ; -- initially examine all members of this object
               If ExamineJSONMembers(v)
                  knownObject = FindMapElement(s_Object(), s_CurMemberKey$)
                  pad = 0
                  validMembers = 0
                  If knownObject
                     While NextJSONMember(v)
                        If FindMapElement(s_Object()\KnownMember(), JSONMemberKey(v))
                           If pad < Len(JSONMemberKey(v))
                              pad = Len(JSONMemberKey(v))
                           EndIf
                           validMembers + 1
                        ElseIf s_Object()\SortUnknownMembers > -1
                           If pad < Len(JSONMemberKey(v))
                              pad = Len(JSONMemberKey(v))
                           EndIf
                           validMembers + 1
                           AddElement(unknownMember$()) : unknownMember$() = JSONMemberKey(v)
                        EndIf
                     Wend
                  ElseIf s_SortUnknownObjects > -1
                     While NextJSONMember(v)
                        If pad < Len(JSONMemberKey(v))
                           pad = Len(JSONMemberKey(v))
                        EndIf
                        validMembers + 1
                        AddElement(unknownMember$()) : unknownMember$() = JSONMemberKey(v)
                     Wend
                  EndIf
                  pad + 2
               EndIf
               
               _WriteLine(key$, "{")
               i = 1
               
               ; -- write known members
               If knownObject
                  ResetList(s_Object()\known())
                  While (i <= validMembers) And NextElement(s_Object()\known())
                     If GetJSONMember(v, s_Object()\known()\s)
                        If i < validMembers : tmp$ = "," : Else : tmp$ = "" : EndIf
                        PushListPosition(s_Object()\known())
                        PushMapPosition(s_Object())
                        s_CurMemberKey$ = JSONMemberKey(v)
                        _TraverseJSON(JSONMemberValue(v), level+1, pad, JSONMemberKey(v), tmp$)
                        PopMapPosition(s_Object())
                        PopListPosition(s_Object()\known())
                        i + 1
                     EndIf
                  Wend
               EndIf
               
               ; -- write unknown members
               If ListSize(unknownMember$()) > 0
                  If knownObject
                     SortList(unknownMember$(), s_Object()\SortUnknownMembers)
                  Else
                     SortList(unknownMember$(), s_SortUnknownObjects)
                  EndIf
                  ForEach unknownMember$()
                     GetJSONMember(v, unknownMember$())
                     If i < validMembers : tmp$ = "," : Else : tmp$ = "" : EndIf
                     s_CurMemberKey$ = JSONMemberKey(v)
                     _TraverseJSON(JSONMemberValue(v), level+1, pad, JSONMemberKey(v), tmp$)
                     i + 1
                  Next
               EndIf
               
               _WriteLine("", "}" + comma$)
            EndIf
            s_CurMemberKey$ = ""
           
         Case #PB_JSON_Array
            last = JSONArraySize(v) - 1
            If last < 0
               _WriteLine(key$, "[]" + comma$)
            Else
               _WriteLine(key$, "[")
               For i = 0 To last-1
                  _TraverseJSON(GetJSONElement(v, i), level+1, pad, "", ",")
               Next
               _TraverseJSON(GetJSONElement(v, last), level+1, pad)
               _WriteLine("", "]" + comma$)
            EndIf
           
         Case #PB_JSON_String
            _WriteLine(key$, #DQUOTE$ + EscapeString(GetJSONString(v)) + #DQUOTE$ + comma$)
           
         Case #PB_JSON_Number
            _WriteLine(key$, GetJSONDouble(v) + comma$)
           
         Case #PB_JSON_Boolean
            If GetJSONBoolean(v) : tmp$ = "true" : Else : tmp$ = "false" : EndIf
            _WriteLine(key$, tmp$ + comma$)
           
         Case #PB_JSON_Null
            _WriteLine(key$, "null" + comma$)
      EndSelect
   EndProcedure
   
   
   Procedure.i Save (json.i, file$, sortUnknown.i=#PB_Sort_Ascending)
      ; -- Save JSON data to a file in the proper format (UTF-8 without BOM);
      ;    pretty-printed, with object members individually arranged or sorted
      ;    according to their names
      ; in : json        : ID of JSON data
      ;      file$       : name of destination file,
      ;                    or "" for output with Debug
      ;      sortUnknown : setting for sorting the members of unknown objects
      ;                    (any PureBasic sort options for strings,
      ;                     or -1 for ignoring unknown objects)
      ; out: return value: 1: file successfully saved,
      ;                    0: file$ = "", or error
      Shared s_SortUnknownObjects, s_Ofn
     
      If IsJSON(json)
         s_SortUnknownObjects = sortUnknown
         
         If file$ <> ""
            s_Ofn = CreateFile(#PB_Any, file$, #PB_UTF8)
            If s_Ofn
               _TraverseJSON(JSONValue(json), 0, 0)
               CloseFile(s_Ofn)
               ProcedureReturn 1
            EndIf
         Else
            CompilerIf #PB_Compiler_Debugger = #False
               CompilerWarning "Enable the Debugger, in order to see the output"
            CompilerEndIf   
            s_Ofn = 0
            _TraverseJSON(JSONValue(json), 0, 0)
         EndIf
      EndIf
     
      ProcedureReturn 0
   EndProcedure
EndModule


CompilerIf #PB_Compiler_IsMainFile
   ; -- Demo
   EnableExplicit
   
   Define.i jn, i, last=5
   Dim input$(last)
   
   input$(0) = "'Hello \'world\''"
   input$(1) = "null"
   input$(2) = "2.7"
   input$(3) = "true"
   input$(4) = "[[4, 3], [1, 2], [5, 6]]"
   input$(5) = "{" +
               "'Given name': 'Mary'," +
               "'Family name': 'Smith'," +
               "'Age': 30," +
               "'Children': {" +
               "'Peter': 6," +
               "'Tom': 2," +
               "'Laura': 5" +
               "}," +
               "'Address': {" +
               "'Country': 'Germany'," +
               "'City': 'Berlin'," +
               "'E-mail': 'mary@smith.de'" +
               "}" +
               "}"
   
   For i = 0 To last
      ReplaceString(input$(i), "'", #DQUOTE$, #PB_String_InPlace)
      jn = ParseJSON(#PB_Any, input$(i))
      If IsJSON(jn) = #False
         Debug "Error, invalid JSON data: " + input$(i)
         End
      EndIf
     
      Debug ComposeJSON(jn, #PB_JSON_PrettyPrint)
      If i = last
         Debug ~"---------  For each object, all member keys are sorted ascending  ---------"
      EndIf   
      JSave::Save(jn, "")
      Debug ""
     
      If i < last
         FreeJSON(jn)
      EndIf   
   Next
   
   Debug "---------  All member keys at basic level individually arranged  ---------"
   JSave::InitObjectStr("", "Given name, Family name, Age, Children, Address")
   JSave::Save(jn, "")
   
   Debug ~"---------  Additionally member keys of object \"Address\" individually arranged  ---------"
   JSave::InitObjectStr("Address", "Country, City, E-mail")
   JSave::Save(jn, "")
CompilerEndIf

_________________
Please excuse my flawed English. My native language is PureBasic.
Search
RSBasic's backups


Last edited by Little John on Thu Feb 15, 2018 6:08 am, edited 2 times in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: Save JSON data with object members well-arranged (Module
PostPosted: Tue Sep 05, 2017 9:25 pm 
Offline
Addict
Addict

Joined: Fri Nov 09, 2012 11:04 pm
Posts: 1516
Location: Uttoxeter, UK
HI Little John,

This is the debug output when run on Mac OS X. Pure Basic 5.50:


Code:
"Hello \"world\""
"Hello \"world\""

null
null

2.7
2.7

true
true

[
  [
    4,
    3
  ],
  [
    1,
    2
  ],
  [
    5,
    6
  ]
]
[
   [
      4,
      3
   ],
   [
      1,
      2
   ],
   [
      5,
      6
   ]
]

{
  "Family name": "Smith",
  "Given name" : "Mary",
  "Address"    : {
      "Country": "Germany",
      "E-mail" : "mary@smith.de",
      "City"   : "Berlin"
    },
  "Children"   : {
      "Laura": 5,
      "Tom"  : 2,
      "Peter": 6
    },
  "Age"        : 30
}
--------------------------------
{
   "Address"    : {
      "City"   : "Berlin",
      "Country": "Germany",
      "E-mail" : "mary@smith.de"
   },
   "Age"        : 30,
   "Children"   : {
      "Laura": 5,
      "Peter": 6,
      "Tom"  : 2
   },
   "Family name": "Smith",
   "Given name" : "Mary"
}

--------------------------------
{
   "Given name" : "Mary",
   "Family name": "Smith",
   "Age"        : 30,
   "Children"   : {
      "Laura": 5,
      "Peter": 6,
      "Tom"  : 2
   },
   "Address"    : {
      "City"   : "Berlin",
      "Country": "Germany",
      "E-mail" : "mary@smith.de"
   }
}
--------------------------------
{
   "Given name" : "Mary",
   "Family name": "Smith",
   "Age"        : 30,
   "Children"   : {
      "Laura": 5,
      "Peter": 6,
      "Tom"  : 2
   },
   "Address"    : {
      "Country": "Germany",
      "City"   : "Berlin",
      "E-mail" : "mary@smith.de"
   }
}


__________________________________________________
Code tags added
06.09.2017
RSBasic

_________________
DE AA EB


Top
 Profile  
Reply with quote  
 Post subject: Re: Save JSON data with object members well-arranged (Module
PostPosted: Wed Sep 06, 2017 10:46 am 
Offline
Addict
Addict
User avatar

Joined: Thu Jun 07, 2007 3:25 pm
Posts: 3145
Location: Berlin, Germany
Hi davido,

this is the correct output. :-)
Thank you very much for testing on Mac!
I have uptaded the text in the first post accordingly.

_________________
Please excuse my flawed English. My native language is PureBasic.
Search
RSBasic's backups


Top
 Profile  
Reply with quote  
 Post subject: Re: Save JSON data with object members well-arranged (Module
PostPosted: Wed Sep 06, 2017 7:59 pm 
Offline
Addict
Addict

Joined: Fri Nov 09, 2012 11:04 pm
Posts: 1516
Location: Uttoxeter, UK
Hi Little John,

Thank you for confirming; I wasn't sure.

I always use JSON because it is so easy to save data.
I had noticed that there was something amiss but I don't compare the saved data just read it. Anyway I couldn't do anything about it. :)
I shall in future use your module.
Thank you for taking the time to compose it and share it. :D

@RSBasic: Sorry for omitting the code tags. Thank you for correcting it - much appreciated.

_________________
DE AA EB


Top
 Profile  
Reply with quote  
 Post subject: Re: Save JSON data with object members well-arranged (Module
PostPosted: Tue Feb 13, 2018 1:47 pm 
Offline
User
User

Joined: Thu Aug 10, 2017 7:35 am
Posts: 65
Thanks for this module, Little John!

Saves a lot of hassle when visually comparing different .jsons
and it makes manual editing also a bit more "foreseeable" :oops:

One thing, though...

As far as I understand you can use JSave::InitObjectStr() multiple
times before you do the save, correct?

If this is true, shouldn't this work:
Code:
  hJSON = CreateJSON(#PB_Any)
  If hJSON
    InsertJSONList(JSONValue(hJSON), Profile())
    JSave::InitObjectStr("", "ID, Position, Enabled, Name, ProfileItems")
    JSave::InitObjectStr("ProfileItems", "Script, Name, Enabled, Position")
    JSave::Save(hJSON, "_converted.json")
    FreeJSON(hJSON)
  EndIf


What I would expect is this:
Code:
[
   {
      "ID"               : "x20160901000000007",
      "Position"         : 7,
      "Enabled"          : 1,
      "Name"             : "Bookmark management",
      "ProfileItems"     : [
         {
            "Script"   : "#ID_4001;",
            "Name"     : "Bookmark item(s)",
            "Enabled"  : 1,
            "Position" : 1
         }
      ]
   },
   {
      "ID"               : "x20160901000000008",
      "Position"         : 8,
      "Enabled"          : 0,
      "Name"             : "Firewall management",
      "ProfileItems"     : [
         {
            "Script"   : "#ID_2001;",
            "Name"     : "Add firewall rule",
            "Enabled"  : 1,
            "Position" : 1
         }
      ]
   }
]


but I get that:
Code:
[
   {
      "ID"               : "x20160901000000007",
      "Position"         : 7,
      "Enabled"          : 1,
      "Name"             : "Bookmark management",
      "ProfileItems"     : [
         {
            "Position" : 1,
            "Enabled"  : 1,
            "Name"     : "Bookmark item(s)",
            "Script"   : "#ID_4001;"
         }
      ]
   },
   {
      "ID"               : "x20160901000000008",
      "Position"         : 8,
      "Enabled"          : 0,
      "Name"             : "Firewall management",
      "ProfileItems"     : [
         {
            "Position" : 1,
            "Enabled"  : 1,
            "Name"     : "Add firewall rule",
            "Script"   : "#ID_2001;"
         }
      ]
   }
]


Top
 Profile  
Reply with quote  
 Post subject: Re: Save JSON data with object members well-arranged (Module
PostPosted: Thu Feb 15, 2018 6:12 am 
Offline
Addict
Addict
User avatar

Joined: Thu Jun 07, 2007 3:25 pm
Posts: 3145
Location: Berlin, Germany
Hi oO0XX0Oo,

thanks for your report!
That was a bug. Please see the first post for the new code.
Can you please tell me whether it works now for you?

_________________
Please excuse my flawed English. My native language is PureBasic.
Search
RSBasic's backups


Top
 Profile  
Reply with quote  
 Post subject: Re: Save JSON data with object members well-arranged (Module
PostPosted: Thu Feb 15, 2018 9:37 am 
Offline
User
User

Joined: Thu Aug 10, 2017 7:35 am
Posts: 65
Thanks for the bugfix, Little John!

The updated code works perfectly fine, the sorting works exactly as expected now :D


Top
 Profile  
Reply with quote  
 Post subject: Re: Save JSON data with object members well-arranged (Module
PostPosted: Thu Feb 15, 2018 1:01 pm 
Offline
Addict
Addict
User avatar

Joined: Sun Nov 05, 2006 11:42 pm
Posts: 4032
Location: Lyon - France
That's works here W7 x86 v5.61 x86
I have apparently the same result than DAVIDO
Thanks for sharing 8)

_________________
ImageThe happiness is a road...
Not a destination


Top
 Profile  
Reply with quote  
 Post subject: Re: Save JSON data with object members well-arranged (Module
PostPosted: Thu Feb 15, 2018 3:22 pm 
Offline
Addict
Addict
User avatar

Joined: Thu Jun 07, 2007 3:25 pm
Posts: 3145
Location: Berlin, Germany
You are welcome!
I am happy that it works fine now.

_________________
Please excuse my flawed English. My native language is PureBasic.
Search
RSBasic's backups


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

 


Powered by phpBB © 2008 phpBB Group
subSilver+ theme by Canver Software, sponsor Sanal Modifiye