JSON encoder and decoder

[mariosk8s]

Got another patch for you.
You're not JSON escaping keys.

Code: Select all

--- a/Json2.pbi 2013-02-19 17:08:17.716628088 +0100                                       
+++ b/Json2.pbi 2013-03-28 16:32:33.984799955 +0100
@@ -253,11 +253,12 @@
           Case 't' ; tabulator
             string + Chr(9)
           Case 'u' ; 4 hex digits
+            *c + SizeOf(CHARACTER) ; eat the 'u'
             If nullByte - *c > 4 * SizeOf(CHARACTER)
               hexDigit = "$"
               ;For i = 1 To 4
                 hexDigit + PeekS(*c, 4)
-                *c + SizeOf(CHARACTER) * 4
+                *c + SizeOf(CHARACTER) * 3 ; only 3 the 4th eaten below
               ;Next
               string + Chr(Val(hexDigit))
             Else
@@ -522,6 +523,18 @@
   EndIf
 EndProcedure
 
+Procedure.s JSON_encodeString(str.s)
+  Protected tmpString.s = ReplaceString(str, "\", "\\") ; \
+  tmpString = ReplaceString(tmpString, Chr(8),  "\b") ; backspace
+  tmpString = ReplaceString(tmpString, Chr(14), "\f") ; formfeed
+  tmpString = ReplaceString(tmpString, Chr(10), "\n")  ; new line
+  tmpString = ReplaceString(tmpString, Chr(13), "\r") ; carriage return
+  tmpString = ReplaceString(tmpString, Chr(9),  "\t") ; tabulator
+  tmpString = ReplaceString(tmpString, Chr(34), "\" + Chr(34)) ; "
+  tmpString = ReplaceString(tmpString, "/",     "\/") ; /
+  ProcedureReturn tmpString
+EndProcedure
+
 Procedure.s JSON_encode(*obj.jsonObj, spaces.i = 0, type.i = #JSON_Type_Undefined)
   Protected tmpString.s
   Protected i.i
@@ -557,17 +570,8 @@
     Case #JSON_Type_Integer
       JSON_addString(Str(*obj\i))
     Case #JSON_Type_String
-      tmpString = ReplaceString(*obj\s, "\", "\\") ; \
-      tmpString = ReplaceString(tmpString, Chr(8), "\b") ; backspace
-      tmpString = ReplaceString(tmpString, Chr(14), "\f") ; formfeed
-      tmpString = ReplaceString(tmpString, Chr(10), "\n")  ; new line
-      tmpString = ReplaceString(tmpString, Chr(13), "\r") ; carriage return
-      tmpString = ReplaceString(tmpString, Chr(9), "\t") ; tabulator
-      tmpString = ReplaceString(tmpString, Chr(34), "\" + Chr(34)) ; "
-      tmpString = ReplaceString(tmpString, "/", "\/") ; /
-      
       JSON_addString(Chr(34))
-      JSON_addString(tmpString)
+      JSON_addString(JSON_encodeString(*obj\s))
       JSON_addString(Chr(34))
     Case #JSON_Type_Array
       JSON_addString("[")
@@ -595,7 +599,7 @@
         NextMapElement(*obj\o())
         JSON_addString(Space(spaces + 2))
         JSON_addString(Chr(34))
-        JSON_addString(MapKey(*obj\o()))
+        JSON_addString(JSON_encodeString(MapKey(*obj\o())))
         JSON_addString(Chr(34))
         JSON_addString(" : ")
         CompilerIf Defined(JSON_UseObjectPointer, #PB_Constant)

This patch includes my previous patch since it hasn't made it into the original post yet.

[PMV]

update 19.04.2013
+ bugfix: encoding for unicode escaped characters (\u) failed
+ bugfix: map keys were not escaped
+ improved: escaping of strings in JSON_encode() now faster
+ improved: test code uses new possibility of CompilerIf #IsMain


Thanks again mariosk8s for bug hunting
I still have not much time for testing, but i hope the new
escape-function is really faster then the previous solution
with ReplaceString().

[Kukulkan]

Thanks PMV. Very good functions!

Kukulkan

[Fangbeast]

Sorry for the beginner question but, when I have a loaded string, how do I list all the available fields if I don't know in advance what they are?

Tried the below but am getting read errors at address 4 starting at "ResetMap(*out\o())"

Code: Select all

; 9780718149789		    the navigator
; 071814788x		      polar shift
; 0552151696		      digital fortress
; 0552149519		      the davinci code
; 0552151769	    pic	deception point
; 0732911206		      the lady of the sorrows

IncludeFile "JsonLib.pb"

Define GoogleJsonFile.i   = OpenFile(#PB_Any, "GoogleBookResults.txt")
Define JsonString.s       = ""
Define FileStringFormat.i = ReadStringFormat(GoogleJsonFile.i)

Define.s URLString.s
Define.s CacheFile.s

If InitNetwork()
  
 URLString.s = "https://www.googleapis.com/books/v1/volumes?projection=full&q=isbn:9780718149789"
 
  CacheFile.s = GetCurrentDirectory() + "GoogleBookResults.txt"

  If URLDownloadToFile_(0, URLString.s, CacheFile.s, 0, 0) = #S_OK
    
    Debug "Success"
    
    While Eof(GoogleJsonFile.i) = #False
      
      JsonString.s + ReadString(GoogleJsonFile.i, FileStringFormat.i) + Chr(13) + Chr(10)
      
    Wend
    
    CloseFile(GoogleJsonFile.i)
    
    Define *out.jsonObj = JSON_decode(JsonString.s)
    
    Debug MapSize(*out\o())
    
    ResetMap(*out\o())
    
    While NextMapElement(*out\o())
      
      Debug MapKey(*out\o())
      
      Debug *out\o()\o("url")\s
      
    Wend

  Else
    
    Debug "Failed"
    
  EndIf
  
Else
  
  End
  
EndIf

**EDIT** Silly me. The cachefile was being truncated at 0 bytes. D'oh! I'll just add a check for filesize.

[USCode]

Looks like C!

Very cool, thanks for sharing!

[Fangbeast]

Okay, I don't know what I am doing but am having trouble reading the output of the below string. I used my routine above to download it from google and I *thought* I could do something simple like: Debug *out\o("textSnippet")\s to show the "textSnippet": " data??

What am I doing wrong?

Code: Select all

{
 "kind": "books#volumes",
 "totalItems": 1,
 "items": [
  {
   "kind": "books#volume",
   "id": "189DPgAACAAJ",
   "etag": "M7ZWVKH/6Gc",
   "selfLink": "https://www.googleapis.com/books/v1/volumes/189DPgAACAAJ",
   "volumeInfo": {
    "title": "The Navigator",
    "subtitle": "A Kurt Austin Adventure, a Novel from the NUMA Files",
    "authors": [
     "Clive Cussler"
    ],
    "publisher": "Michael Joseph",
    "publishedDate": "2007-05-20",
    "description": "Years ago, an ancient Phoenician statue known as the Navigator was stolen from the Baghdad Museum, and there are men who would do anything to get their hands on it. The first victim is a crooked antiquities dealer, murdered in cold blood. Their second very nearly is a UN investigator who, were it not for the timely assistance of Austin and Zavala, would now be at the bottom of a very watery grave. What's so special about this statue? Austin wonders. The search for answers will take the NUMA team on an astonishing odyssey through time and space, one that encompasses no less than the lost treasures of King Solomon, a mysterious packet of documents personally encoded by Thomas Jefferson, and a top secret scientific project that could change the world forever. And that's before the surprises really begin . . .",
    "industryIdentifiers": [
     {
      "type": "ISBN_10",
      "identifier": "0718149785"
     },
     {
      "type": "ISBN_13",
      "identifier": "9780718149789"
     }
    ],
    "pageCount": 437,
    "printType": "BOOK",
    "averageRating": 3.0,
    "ratingsCount": 27,
    "contentVersion": "preview-1.0.0",
    "imageLinks": {
     "smallThumbnail": "http://bks7.books.google.com.au/books?id=189DPgAACAAJ&printsec=frontcover&img=1&zoom=5&source=gbs_api",
     "thumbnail": "http://bks7.books.google.com.au/books?id=189DPgAACAAJ&printsec=frontcover&img=1&zoom=1&source=gbs_api"
    },
    "language": "en",
    "previewLink": "http://books.google.com.au/books?id=189DPgAACAAJ&dq=isbn:9780718149789&hl=&cd=1&source=gbs_api",
    "infoLink": "http://books.google.com.au/books?id=189DPgAACAAJ&dq=isbn:9780718149789&hl=&source=gbs_api",
    "canonicalVolumeLink": "http://books.google.com.au/books/about/The_Navigator.html?hl=&id=189DPgAACAAJ"
   },
   "saleInfo": {
    "country": "AU",
    "saleability": "NOT_FOR_SALE",
    "isEbook": false
   },
   "accessInfo": {
    "country": "AU",
    "viewability": "NO_PAGES",
    "embeddable": false,
    "publicDomain": false,
    "textToSpeechPermission": "ALLOWED",
    "epub": {
     "isAvailable": false
    },
    "pdf": {
     "isAvailable": false
    },
    "webReaderLink": "http://books.google.com.au/books/reader?id=189DPgAACAAJ&hl=&printsec=frontcover&output=reader&source=gbs_api",
    "accessViewStatus": "NONE"
   },
   "searchInfo": {
    "textSnippet": "Years ago, an ancient Phoenician statue known as the Navigator was stolen from the Baghdad Museum, and there are men who would do anything to get their hands on it."
   }
  }
 ]
}

[StarBootics]

Hello Fangbeast,

I have play a little with your code and I finally make it to work. Since I'm on linux I have loaded the json file directly from my computer not over the internet but it should work.

Code: Select all

; 9780718149789          the navigator
; 071814788x            polar shift
; 0552151696            digital fortress
; 0552149519            the davinci code
; 0552151769       pic   deception point
; 0732911206            the lady of the sorrows

IncludeFile "JsonLib.pb"

; Define GoogleJsonFile.i   = OpenFile(#PB_Any, "GoogleBookResults.txt")
; Define JsonString.s       = ""
; Define FileStringFormat.i = ReadStringFormat(GoogleJsonFile.i)

Define.s URLString.s
Define.s CacheFile.s

; If InitNetwork()
; 	
; 	URLString.s = "https://www.googleapis.com/books/v1/volumes?projection=full&q=isbn:9780718149789"
; 	
; 	CacheFile.s = GetCurrentDirectory() + "GoogleBookResults.txt"

; 	If URLDownloadToFile_(0, URLString.s, CacheFile.s, 0, 0) = #S_OK
If ReadFile(0, "book.json")
  Debug "Success"
  
  While Eof(GoogleJsonFile.i) = #False
    
    JsonString.s + ReadString(GoogleJsonFile.i, FileStringFormat.i) + Chr(13) + Chr(10)
    
  Wend
  
  CloseFile(GoogleJsonFile.i)
  
  Define *out.jsonObj = JSON_decode(JsonString.s)

  Debug "MapSize : " + Str(MapSize(*out\o()))
  
  ResetMap(*out\o())
  
  While NextMapElement(*out\o())
    
    JSON_Debug(*out.jsonObj, MapKey(*out\o()))
    
   ; Debug MapKey(*out\o()) + " : " + *out\o()\o("url")\s
    
  Wend
  
Else
  
  Debug "Failed"
  
EndIf

; Else
; 	
; 	End
; 	
; EndIf

Then the debugger output :

Code: Select all

Success
MapSize : 3
totalItems (object) : {
totalItems(int) : 1
kind (string) : books#volumes
items (array) : [
1. (object) : {
searchInfo (object) : {
textSnippet (string) : Years ago, an ancient Phoenician statue known as the Navigator was stolen from the Baghdad Museum, and there are men who would do anything to get their hands on it.
}
volumeInfo (object) : {
language (string) : en
averageRating (float) : 3
canonicalVolumeLink (string) : http://books.google.com.au/books/about/The_Navigator.html?hl=&id=189DPgAACAAJ
description (string) : Years ago, an ancient Phoenician statue known as the Navigator was stolen from the Baghdad Museum, and there are men who would do anything to get their hands on it. The first victim is a crooked antiquities dealer, murdered in cold blood. Their second very nearly is a UN investigator who, were it not for the timely assistance of Austin and Zavala, would now be at the bottom of a very watery grave. What's so special about this statue? Austin wonders. The search for answers will take the NUMA team on an astonishing odyssey through time and space, one that encompasses no less than the lost treasures of King Solomon, a mysterious packet of documents personally encoded by Thomas Jefferson, and a top secret scientific project that could change the world forever. And that's before the surprises really begin . . .
subtitle (string) : A Kurt Austin Adventure, a Novel from the NUMA Files
title (string) : The Navigator
publishedDate (string) : 2007-05-20
ratingsCount(int) : 27
infoLink (string) : http://books.google.com.au/books?id=189DPgAACAAJ&dq=isbn:9780718149789&hl=&source=gbs_api
previewLink (string) : http://books.google.com.au/books?id=189DPgAACAAJ&dq=isbn:9780718149789&hl=&cd=1&source=gbs_api
pageCount(int) : 437
printType (string) : BOOK
imageLinks (object) : {
smallThumbnail (string) : http://bks7.books.google.com.au/books?id=189DPgAACAAJ&printsec=frontcover&img=1&zoom=5&source=gbs_api
thumbnail (string) : http://bks7.books.google.com.au/books?id=189DPgAACAAJ&printsec=frontcover&img=1&zoom=1&source=gbs_api
}
industryIdentifiers (array) : [
1. (object) : {
type (string) : ISBN_10
identifier (string) : 0718149785
}
2. (object) : {
type (string) : ISBN_13
identifier (string) : 9780718149789
}
]
publisher (string) : Michael Joseph
authors (array) : [
1. (string) : Clive Cussler
]
contentVersion (string) : preview-1.0.0
}
selfLink (string) : https://www.googleapis.com/books/v1/volumes/189DPgAACAAJ
accessInfo (object) : {
country (string) : AU
textToSpeechPermission (string) : ALLOWED
pdf (object) : {
isAvailable (false)
}
epub (object) : {
isAvailable (false)
}
accessViewStatus (string) : NONE
viewability (string) : NO_PAGES
publicDomain (false)
embeddable (false)
webReaderLink (string) : http://books.google.com.au/books/reader?id=189DPgAACAAJ&hl=&printsec=frontcover&output=reader&source=gbs_api
}
kind (string) : books#volume
saleInfo (object) : {
isEbook (false)
country (string) : AU
saleability (string) : NOT_FOR_SALE
}
id (string) : 189DPgAACAAJ
etag (string) : M7ZWVKH/6Gc
}
]
}

Now how you can use information just loaded it's a different story but if you take the time to study how the JSON_Debug() work I'm pretty sure someone like you will be able to do it.

Regards
StarBootics

[Fangbeast]

Thanks StarBootics, studying the json lib, I didn't really understand it, that's why I had no idea how/why to get the output. Thanks for the pointer to the right direction.

I'm missing something though. How do I get access directly to my elements with the type in the loop as shown here in the original jsonlib page.

Code: Select all

Debug "direct Access:"
Debug "Nummer: " + *out\o("Nummer")\s
Debug "Vorname: " + *out\o("Inhaber")\o("Vorname")\s
Debug "Vorliebe 2: " + *out\o("Inhaber")\o("Vorlieben")\a(2)\s
Debug " "

I should be able to do this?

Debug "Text snippet: " + *out\o("textSnippet")\s

but nothing is returned

[PMV]

The "textSnippet" is part of the "items"-Array, isn't it?
So you need to go through it first

it should be something like this then:

Code: Select all

Debug *out\o("items")\a(0)\o("textSnippet")\s

MFG PMV

[Fangbeast]

The "textSnippet" is part of the "items"-Array, isn't it?
So you need to go through it first

it should be something like this then:

Code: Select all

Debug *out\o("items")\a(0)\o("textSnippet")\s

MFG PMV

Sorry to be so thick, that didn't work. I find it hard to follow code where every variable is a single letter. If I understood it, I would use long variables that I could understand (grin), in the tradition of PureBasic.

This bit looks like ASCII art (*out\o("items")\a(0)\o)!!

[PMV]

Normally, i just write variables as long as possible, but in this case,
it would result in much to long lines. I prefere lines with 70 chars instead
of 700 But it is just an include, you can make your own
one by replacing the letters with the complete words.
If you understand JSON, it shouldn't be unpossible
o = object
a = array
s = string
... and so on

MFG PMV

[Fangbeast]

Normally, i just write variables as long as possible, but in this case,
it would result in much to long lines. I prefere lines with 70 chars instead
of 700 But it is just an include, you can make your own
one by replacing the letters with the complete words.
If you understand JSON, it shouldn't be unpossible
o = object
a = array
s = string
... and so on
MFG PMV

Hohoho, a comedian yet. No, I don't understand json and no idea what I am doing wrong LOL. But thanks for the suggestion:):)

[PMV]

The "textSnippet" is part of the "items"-Array, isn't it?
So you need to go through it first

it should be something like this then:

Code: Select all

Debug *out\o("items")\a(0)\o("textSnippet")\s

MFG PMV

Sorry to be so thick, that didn't work.

Of course, because there is another object surrounding the textSnippet
Right is this:

Code: Select all

*out\o("items")\a(0)\o("searchInfo")\o("textSnippet")\s

1. root element (*out)
2. object "items" that contains an array
3. array has just one element, so index is 0
4. object "SearchInfo" that has just one
5. object in it that is your "textSnippet"
6. access the data, in this case it is a string so it is the structure-field "s"

MFG PMV

[Fangbeast]

I bow to your munificense and help. My tiny brain is happy! I should bowwow an animated graphic from KCC as thanks to you:):)

Bugger! I am getting a syntax error with that line. I forgot to put something in front of it LOL!

Fixed!

D'oh!! I finally thought to capture the file from the browser instead of the program. Finally I can see from the indenting which section belongs to which object!

[Fangbeast]

One last question please PMV. This section

Code: Select all

epub (object) : {
isAvailable (false)
}

I thought I could get it like this :

Code: Select all

Debug "epub is available:     " + *out\o("items")\a(0)\o("accessInfo")\o("epub")\o("isAvailable")\s

But no response as it is a true/false return (not a string \s type) and I don't know how to handle it?

Otherwise I have all the other fields coming back now (Phew!)