Write/save structure to disk?

Fangbeast · Post by **Fangbeast** » Tue Oct 06, 2015 5:11 am

Did a forum search on "Write structure" and also "Save structure" to disk, came up with 30 possible pages and I know I saw it somewhere before...

Given a structure as below and a linked list based on it, is there a way to easily save the elements of each line in the list without having to individually specify them?

Code: Select all

Structure MasterlogData
  Frequency.s           ;  1
  Mode.s                ;  2
  Caller.s              ;  3
  Slash.s               ;  4
  Called.s              ;  5
  Rx_RPT1.s             ;  6
  Rx_RPT2.s             ;  7
  Message.s             ;  8
  Status.s              ;  9
  Received_date.s       ; 10
  BK.s                  ; 11
  EMR.s                 ; 12
  Latitude.s            ; 13
  Longitude.s           ; 14
  Altitude.s            ; 15
  SSID.s                ; 16
  D_PRS_Symbol.s        ; 17
  Course.s              ; 18
  Speed.s               ; 19
  Power.s               ; 20
  Height.s              ; 21
  Gain.s                ; 22
  Directivity.s         ; 23
  Object_Item_Name.s    ; 24
  Data_Type.s           ; 25
  Temperature.s         ; 26
  Rainfall.s            ; 27
  Rainfall_24_Hours.s   ; 28
  Rainfall_Midnight.s   ; 29
  Wind_Direction.s      ; 20
  Wind_Speed.s          ; 21
  Gust_Speed.s          ; 22
  Barometric.s          ; 23
  Humidity.s            ; 24
  GPS_Time_Stamp.s      ; 25
  GPS_Message.s         ; 26
EndStructure

Global NewList MasterLog.MasterlogData()

At the moment, I have to do a:

Code: Select all

ForEach MasterLog()
  WriteString(MyFileHandle, MasterLog()\Frequency)
  ; then the next element, etc, etc, etc
Next

Is there a faster way?

Demivec · Post by **Demivec** » Tue Oct 06, 2015 5:30 am

Perhaps using JSON. Example Link.

Fangbeast · Post by **Fangbeast** » Tue Oct 06, 2015 6:33 am

Demivec wrote:Perhaps using JSON. Example Link.

Thanks Demivec, that's interesting. Don't know how to work with json yet so that will be for later.

Will keep searching, i know there was an example somewhere in the forum..

Keya · Post by **Keya** » Tue Oct 06, 2015 7:19 am

hiya Fangbeast please forigve me if im misunderstanding your question!
but i dont think there is a much easier way than iterating each item. If it was a single solid structure you could just use @ to get address, SizeOf to get size, and WriteData to file, but because strings in a structure are just pointers it's not a single solid structure as you probably know! Perhaps you could test to see if the locations of all the pointed strings are in a contiguous memory so that the first and last items are also first and last in memory, and with all other items inbetween... then you could just call WriteData twice, but i dont think it would be contiguous because it needs to allow for dynamic resizing of strings so they could get relocated? i stop thinking now

User_Russian · Post by **User_Russian** » Tue Oct 06, 2015 7:31 am

Fangbeast wrote:Don't know how to work with json yet so that will be for later.

Code: Select all

Structure MasterlogData
  Frequency.s           ;  1
  Mode.s                ;  2
  Caller.s              ;  3
  Slash.s               ;  4
  Called.s              ;  5
  Rx_RPT1.s             ;  6
  Rx_RPT2.s             ;  7
  Message.s             ;  8
  Status.s              ;  9
  Received_date.s       ; 10
  BK.s                  ; 11
  EMR.s                 ; 12
  Latitude.s            ; 13
  Longitude.s           ; 14
  Altitude.s            ; 15
  SSID.s                ; 16
  D_PRS_Symbol.s        ; 17
  Course.s              ; 18
  Speed.s               ; 19
  Power.s               ; 20
  Height.s              ; 21
  Gain.s                ; 22
  Directivity.s         ; 23
  Object_Item_Name.s    ; 24
  Data_Type.s           ; 25
  Temperature.s         ; 26
  Rainfall.s            ; 27
  Rainfall_24_Hours.s   ; 28
  Rainfall_Midnight.s   ; 29
  Wind_Direction.s      ; 20
  Wind_Speed.s          ; 21
  Gust_Speed.s          ; 22
  Barometric.s          ; 23
  Humidity.s            ; 24
  GPS_Time_Stamp.s      ; 25
  GPS_Message.s         ; 26
EndStructure

Global NewList MasterLog.MasterlogData()

AddElement(MasterLog())
AddElement(MasterLog())


If CreateJSON(0)
  x = JSONValue(0)
  If x
    InsertJSONList(x, MasterLog())
    SaveJSON(0, "D:\Test.json")
  EndIf
  FreeJSON(0)
EndIf

Fangbeast · Post by **Fangbeast** » Tue Oct 06, 2015 7:36 am

Keya wrote:hiya Fangbeast please forigve me if im misunderstanding your question!
but i dont think there is a much easier way than iterating each item. If it was a single solid structure you could just use @ to get address, SizeOf to get size, and WriteData to file, but because strings in a structure are just pointers it's not a single solid structure as you probably know! Perhaps you could test to see if the locations of all the pointed strings are in a contiguous memory so that the first and last items are also first and last in memory, and with all other items inbetween... then you could just call WriteData twice, but i dont think it would be contiguous because it needs to allow for dynamic resizing of strings so they could get relocated? i stop thinking now

In other words, complicated

:):)

Fangbeast · Post by **Fangbeast** » Tue Oct 06, 2015 7:37 am

User_Russian wrote:

Fangbeast wrote:Don't know how to work with json yet so that will be for later.

Code: Select all

Structure MasterlogData
  Frequency.s           ;  1
  Mode.s                ;  2
  Caller.s              ;  3
  Slash.s               ;  4
  Called.s              ;  5
  Rx_RPT1.s             ;  6
  Rx_RPT2.s             ;  7
  Message.s             ;  8
  Status.s              ;  9
  Received_date.s       ; 10
  BK.s                  ; 11
  EMR.s                 ; 12
  Latitude.s            ; 13
  Longitude.s           ; 14
  Altitude.s            ; 15
  SSID.s                ; 16
  D_PRS_Symbol.s        ; 17
  Course.s              ; 18
  Speed.s               ; 19
  Power.s               ; 20
  Height.s              ; 21
  Gain.s                ; 22
  Directivity.s         ; 23
  Object_Item_Name.s    ; 24
  Data_Type.s           ; 25
  Temperature.s         ; 26
  Rainfall.s            ; 27
  Rainfall_24_Hours.s   ; 28
  Rainfall_Midnight.s   ; 29
  Wind_Direction.s      ; 20
  Wind_Speed.s          ; 21
  Gust_Speed.s          ; 22
  Barometric.s          ; 23
  Humidity.s            ; 24
  GPS_Time_Stamp.s      ; 25
  GPS_Message.s         ; 26
EndStructure

Global NewList MasterLog.MasterlogData()

AddElement(MasterLog())
AddElement(MasterLog())


If CreateJSON(0)
  x = JSONValue(0)
  If x
    InsertJSONList(x, MasterLog())
    SaveJSON(0, "D:\Test.json")
  EndIf
  FreeJSON(0)
EndIf

Thank you, that's simpler than what I was trying to do. This little ID-51A log parser is very useful to me now that I can do all these things with it.

ostapas · Post by **ostapas** » Tue Oct 06, 2015 9:49 am

Also, if the list gets extremely large, you can compress it before saving:

Code: Select all

Structure MasterlogData
  Frequency.s           ;  1
  Mode.s                ;  2
  Caller.s              ;  3
  Slash.s               ;  4
  Called.s              ;  5
  Rx_RPT1.s             ;  6
  Rx_RPT2.s             ;  7
  Message.s             ;  8
  Status.s              ;  9
  Received_date.s       ; 10
  BK.s                  ; 11
  EMR.s                 ; 12
  Latitude.s            ; 13
  Longitude.s           ; 14
  Altitude.s            ; 15
  SSID.s                ; 16
  D_PRS_Symbol.s        ; 17
  Course.s              ; 18
  Speed.s               ; 19
  Power.s               ; 20
  Height.s              ; 21
  Gain.s                ; 22
  Directivity.s         ; 23
  Object_Item_Name.s    ; 24
  Data_Type.s           ; 25
  Temperature.s         ; 26
  Rainfall.s            ; 27
  Rainfall_24_Hours.s   ; 28
  Rainfall_Midnight.s   ; 29
  Wind_Direction.s      ; 20
  Wind_Speed.s          ; 21
  Gust_Speed.s          ; 22
  Barometric.s          ; 23
  Humidity.s            ; 24
  GPS_Time_Stamp.s      ; 25
  GPS_Message.s         ; 26
EndStructure

UseZipPacker()  
NewList MasterLog.MasterlogData()

AddElement(MasterLog())
MasterLog()\Altitude = "123"
AddElement(MasterLog())
MasterLog()\Altitude = "321"

;Save 
lJSON = CreateJSON(#PB_Any)
If lJSON
  InsertJSONList(JSONValue(lJSON), MasterLog())
  lJSONSize = ExportJSONSize(lJSON)
  *lJSONBuffer = AllocateMemory(lJSONSize)
  *lJSONBufferCompressed = AllocateMemory(lJSONSize)
  ExportJSON(lJSON, *lJSONBuffer, lJSONSize)
  lJSONCompressedSize = CompressMemory(*lJSONBuffer, lJSONSize, *lJSONBufferCompressed, lJSONSize, #PB_PackerPlugin_Zip)
  lFileJSON = CreateFile(#PB_Any, GetCurrentDirectory() + "Data.dat") 
  WriteData(lFileJSON, *lJSONBufferCompressed, lJSONCompressedSize)
  CloseFile(lFileJSON)
  FreeJSON(lJSON)
  FreeMemory(*lJSONBuffer)
  FreeMemory(*lJSONBufferCompressed )
EndIf   

FreeList(MasterLog())
NewList MasterLog.MasterlogData()

;Load
lFileSize = FileSize(GetCurrentDirectory() + "Data.dat")
If lFileSize
  lFile = ReadFile(#PB_Any, GetCurrentDirectory() + "Data.dat")
  If lFile 
    *lJSONBufferCompressed = AllocateMemory(lFileSize)
    *lJSONBuffer = AllocateMemory(lFileSize*60)        
    ReadData(lFile, *lJSONBufferCompressed, lFileSize)
    lUncompressedSize = UncompressMemory(*lJSONBufferCompressed, lFileSize, *lJSONBuffer, lFileSize*60, #PB_PackerPlugin_Zip)
    lJSON = CatchJSON(#PB_Any, *lJSONBuffer, lUncompressedSize)
    FreeMemory(*lJSONBuffer)
    FreeMemory(*lJSONBufferCompressed)
    CloseFile(lFile)
    ExtractJSONList(JSONValue(lJSON), MasterLog())  
  EndIf
EndIf

ForEach MasterLog()
  Debug MasterLog()\Altitude  
Next

Fangbeast · Post by **Fangbeast** » Tue Oct 06, 2015 12:04 pm

ostapas wrote:Also, if the list gets extremely large, you can compress it before saving:

Code: Select all

UseZipPacker()  
NewList MasterLog.MasterlogData()

AddElement(MasterLog())
MasterLog()\Altitude = "123"
AddElement(MasterLog())
MasterLog()\Altitude = "321"

;Save 
lJSON = CreateJSON(#PB_Any)
If lJSON
  InsertJSONList(JSONValue(lJSON), MasterLog())
  lJSONSize = ExportJSONSize(lJSON)
  *lJSONBuffer = AllocateMemory(lJSONSize)
  *lJSONBufferCompressed = AllocateMemory(lJSONSize)
  ExportJSON(lJSON, *lJSONBuffer, lJSONSize)
  lJSONCompressedSize = CompressMemory(*lJSONBuffer, lJSONSize, *lJSONBufferCompressed, lJSONSize, #PB_PackerPlugin_Zip)
  lFileJSON = CreateFile(#PB_Any, GetCurrentDirectory() + "Data.dat") 
  WriteData(lFileJSON, *lJSONBufferCompressed, lJSONCompressedSize)
  CloseFile(lFileJSON)
  FreeJSON(lJSON)
EndIf   

FreeList(MasterLog())
NewList MasterLog.MasterlogData()

;Load
lFileSize = FileSize(GetCurrentDirectory() + "Data.dat")
If lFileSize
  lFile = ReadFile(#PB_Any, GetCurrentDirectory() + "Data.dat")
  If lFile 
    *lJSONBufferCompressed = AllocateMemory(lFileSize)
    *lJSONBuffer = AllocateMemory(lFileSize*60)        
    ReadData(lFile, *lJSONBufferCompressed, lFileSize)
    lUncompressedSize = UncompressMemory(*lJSONBufferCompressed, lFileSize, *lJSONBuffer, lFileSize*60, #PB_PackerPlugin_Zip)
    lJSON = CatchJSON(#PB_Any, *lJSONBuffer, lUncompressedSize)
    FreeMemory(*lJSONBuffer)
    FreeMemory(*lJSONBufferCompressed)
    CloseFile(lFile)
    ExtractJSONList(JSONValue(lJSON), MasterLog())  
  EndIf
EndIf

ForEach MasterLog()
  Debug MasterLog()\Altitude  
Next

That's pretty impresive ostapas. No way my tiny brain would have ever thought of that.

When I do a raw log dump, the file is 375K. When I do an xml or json dump as shown, the log is just shy of 3 meg so your compression idea is great. Thank you.

Lebostein · Post by **Lebostein** » Wed Mar 01, 2017 10:20 am

Are there alternatives to save/load a complex structure (in a binary way for example)? JSON text parsing and writing is extremely slow....

Would be nice to have such functions like StructureToMemory() or StructureFromMemory() to transfer a structure to a single memory block (and vice versa) which you can compress, encrypt and/or write to disc...

PS: like the "pickle" module in Python (https://docs.python.org/3/library/pickl ... ule-pickle)

There are fundamental differences between the pickle protocols and JSON (JavaScript Object Notation):
JSON is a text serialization format (it outputs unicode text, although most of the time it is then encoded to utf-8), while pickle is a binary serialization format;
JSON is human-readable, while pickle is not;
JSON is interoperable and widely used outside of the Python ecosystem, while pickle is Python-specific;
JSON, by default, can only represent a subset of the Python built-in types, and no custom classes; pickle can represent an extremely large number of Python types (many of them automatically, by clever usage of Python’s introspection facilities; complex cases can be tackled by implementing specific object APIs).

skywalk · Post by **skywalk** » Wed Mar 01, 2017 2:41 pm

All the elements in the structure could be expressed as doubles and integers. Only commit to string when displaying the data in tabular format. Much faster and smaller log.

wilbert · Post by **wilbert** » Wed Mar 01, 2017 5:52 pm

You might consider using fixed strings in your structure like
Latitude.s{8}
That would make saving and loading very easy and fast.

Lunasole · Post by **Lunasole** » Thu Mar 02, 2017 4:46 am

Lebostein wrote:Are there alternatives to save/load a complex structure (in a binary way for example)? JSON text parsing and writing is extremely slow....

I just hate JSON (even for its name itself ^_^), so typically using XML for that which is 1000 times better as for me.
Here is some ready-to-use code, however not sure will it be fast enough for your case
http://www.purebasic.fr/english/viewtop ... 12&t=67110

Previously I was using plain binary to store all the stuff, that's better for performance and memory consumption, but really becomes painful if you writing that import/export for everything manually and having large nested structure with strings and so on. Have no any idea how it is possible to automate that "forgotten art of structure packing" nicely.

Post by **idle** » Thu Mar 02, 2017 6:22 am

Fangles just use fixed size strings in your structure like you would when you create an sql table

Code: Select all

Structure MasterlogData
  Frequency.s{60}       ;  1     
  Mode.s{60}                ;  2
  Caller.s{60}              ;  3
  Slash.s{60}               ;  4
  Called.s{60}              ;  5
  Rx_RPT1.s{60}             ;  6
  Rx_RPT2.s{60}             ;  7
  Message.s{60}             ;  8
  Status.s{60}              ;  9
  Received_date.s{60}       ; 10
  BK.s{60}                  ; 11
  EMR.s{60}                 ; 12
  Latitude.s{60}            ; 13
  Longitude.s{60}           ; 14
  Altitude.s{60}            ; 15
  SSID.s{60}                ; 16
  D_PRS_Symbol.s{60}        ; 17
  Course.s{60}              ; 18
  Speed.s{60}               ; 19
  Power.s{60}               ; 20
  Height.s{60}              ; 21
  Gain.s{60}                ; 22
  Directivity.s{60}         ; 23
  Object_Item_Name.s{60}    ; 24
  Data_Type.s{60}           ; 25
  Temperature.s{60}         ; 26
  Rainfall.s{60}            ; 27
  Rainfall_24_Hours.s{60}   ; 28
  Rainfall_Midnight.s{60}   ; 29
  Wind_Direction.s{60}      ; 20
  Wind_Speed.s{60}          ; 21
  Gust_Speed.s{60}          ; 22
  Barometric.s{60}          ; 23
  Humidity.s{60}            ; 24
  GPS_Time_Stamp.s{60}      ; 25
  GPS_Message.s{60}         ; 26
EndStructure

Debug SizeOf(MasterlogData) 

Global NewList MasterLog.MasterlogData()
 
; Code:
 ForEach MasterLog()
   WriteData(MyFileHandle, @MasterLog(),SizeOf(MasterlogData))
  Next

Fangbeast · Post by **Fangbeast** » Thu Mar 02, 2017 7:38 am

Fangles just use fixed size strings in your structure like you would when you create an sql table

Idle, these posts have nothing to do with me since 2015!!

PureBasic Forums - English

Write/save structure to disk?

Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?

Re: Write/save structure to disk?