Best way to load/save plain text files easily with Lists in different projects? (or pass a List back from a procedure)

[Nudgy]

Hi,

I am new to PureBasic, coming from other programming languages. I will be needing to open, read and manipulate various plain text files, some of them many MB in size, and it seems that Lists would be a good option for this. However, it seems that there is no built in function to load Lists directly from files, so I would have to write something with ReadString and WriteString.

To increase my efficiency when working in different projects, I would like to make a single, generic procedure (e.g. "LoadListFromFile(List myList.s)" which I could include in a PBI file and call whenever I need to load a List of strings.

PB can pass a List to a Procedure via e.g. Procedure MyProcedureName(List myList.s). But is there an elegant way for the procedure to "pass" the loaded List contents back to whichever List was passed to the procedure, for example in ProcedureReturn? If not, the only option that I can think of is to (1) use my Procedure to load the text file and (2) fill a generic "global" List, then (3) have my original List read from that other list in a separate step.

I hope my question makes sense. Looking forward to your advice!

Thanks.

[PBJim]

I am new to PureBasic, coming from other programming languages. I will be needing to open, read and manipulate various plain text files, some of them many MB in size, and it seems that Lists would be a good option for this. However, it seems that there is no built in function to load Lists directly from files, so I would have to write something with ReadString and WriteString.

Dear Nudgy, welcome to the forum.

It's straightforward to populate a list from the contents of a text file. Depending on the format of the text file, a read of each line — if the lines are CR/LF separated — then just add each line to a list element, with only two statements :

Code: Select all

AddElement(TextList())                                      ; Create new element in the list
TextList() = mytext.s                                       ; Add each to history list

Is your text file in a pre-existing form, or do you have the liberty to create the text file to your own format? If you can create it yourself, I recommend taking a look at PureBasic's JSON functions, as they serve as a very fast and convenient way to save and load a list, to and from a file. I use LoadJSON and ExtractJSONList(). They take a bit of working out at first, to get it right, but we can point you in the right direction if you're stuck.

PB can pass a List to a Procedure via e.g. Procedure MyProcedureName(List myList.s). But is there an elegant way for the procedure to "pass" the loaded List contents back to whichever List was passed to the procedure, for example in ProcedureReturn?

Lists and maps can be passed as procedure parameters and their contents then modified within the called procedure. Unlike conventional variables, the list is the same list within the procedure, as it is outside the procedure. Like this :

Code: Select all

Procedure MyUpdateProc(List Mylist.s())
  AddElement(MyList())
  MyList() = "ghi"
  AddElement(MyList())
  MyList() = "jkl"
EndProcedure

NewList TextList.s()

AddElement(TextList())
TextList() = "abc"
AddElement(TextList())
TextList() = "def"

MyUpdateProc(TextList())

ForEach TextList()
  Debug TextList()
Next

[infratec]

If you have a list with a structure, you can save it as CSV file or as a JSON file.

SaveJSON() LoadJSON() in combination with ExtractJSONStructure(), InsertJSONStructure()

You can 'return' a list with the list as parameter.

[infratec]

Example:

Code: Select all

EnableExplicit

Structure TestListStructure
  Name$
  Value.i
EndStructure


Procedure.i TestProcedure(List TestList.TestListStructure())
  
  Protected.i i
  
  For i = 1 To 10
    AddElement(TestList())
    TestList()\Name$ = "Name_" + Str(i)
    TestList()\Value = i
  Next i
  
  ProcedureReturn ListSize(TestList())
  
EndProcedure



Define JSON.i
Define JSON$
NewList TestList.TestListStructure()


Debug 1

TestProcedure(TestList())

ForEach TestList()
  Debug TestList()\Name$ + " = " + Str(TestList()\Value)
Next

JSON = CreateJSON(#PB_Any)
If JSON
  InsertJSONList(JSONValue(JSON), TestList())
  SaveJSON(JSON, ProgramFilename() + ".json")
  FreeJSON(JSON)
EndIf


Debug 2

ClearList(TestList())

ForEach TestList()
  Debug TestList()\Name$ + " = " + Str(TestList()\Value)
Next

Debug 3

JSON = LoadJSON(#PB_Any, ProgramFilename() + ".json")
If JSON
  
  ExtractJSONList(JSONValue(JSON), TestList())
  
  ForEach TestList()
    Debug TestList()\Name$ + " = " + Str(TestList()\Value)
  Next
  
  FreeJSON(JSON)
EndIf

If MessageRequester("Question", "Delete" + #LF$ + ProgramFilename() + ".json" + #LF$ + "?", #PB_MessageRequester_YesNo) = #PB_MessageRequester_Yes
  If FileSize(ProgramFilename() + ".json") > 0
    DeleteFile(ProgramFilename() + ".json")
  EndIf
EndIf

[Nudgy]

Is your text file in a pre-existing form, or do you have the liberty to create the text file to your own format? If you can create it yourself, I recommend taking a look at PureBasic's JSON functions, as they serve as a very fast and convenient way to save and load a list, to and from a file. I use LoadJSON and ExtractJSONList(). They take a bit of working out at first, to get it right, but we can point you in the right direction if you're stuck.

Thanks for your quick reply. My files for now are simple CSV and tab separated files, with one record per line. I haven't studied JSON much yet, but to me the JSON format seems like it is quite "verbose and would be slow to parse in practice for bigger files. But maybe your experience is different?

Lists and maps can be passed as procedure parameters and their contents then modified within the called procedure. Unlike conventional variables, the list is the same list within the procedure, as it is outside the procedure.

Thank you - this is good to know! My assumption was that the resulting list had to be returned somehow, but it seems that you are right, and that if you pass a list as a parameter to Procedure, that list itself can be edited directly in the Procedure. Based on your input I managed to create a procedure which seems to work for me for loading lists via an "Include" file. I might need to do some more testing, but this is what I used, in case anyone is interested:

Code: Select all

Procedure.b StringListLoadFile(List lListInput.s(), FileName.s)
    Protected line.s, pos.i
  
  ; Check if the file exists
  If FileSize(fileName) <> -1
    If ReadFile(0, fileName) ; If the file exists, open it for reading
      
      ClearList(lListInput()) ; First clear the list so we don't add to any existing entries in the List
      
      While Eof(0) = 0
        line = ReadString(0) ; Read the line
        AddElement(lListInput()) ; Create a list element
        lListInput() = line ; Populate the list element
      Wend
      CloseFile(0)
      ProcedureReturn #True
    Else
      ProcedureReturn #False ; Return false if error
    EndIf
  Else
    ProcedureReturn #False ; Return false if error
  EndIf
  
EndProcedure

[Nudgy]

If you have a list with a structure, you can save it as CSV file or as a JSON file.

SaveJSON() LoadJSON() in combination with ExtractJSONStructure(), InsertJSONStructure()

You can 'return' a list with the list as parameter.

Thank you for this and for the example. I will have to study this closer, since I am not used to working with JSON files. But it seems at least I have a solution now for the CSV files (see my other reply) - and some JSON "homework" to do

[infratec]

For large lists, a CSV file needs less space on disk.
But ...
you maybe will run into the problem that PB is slow in string operations.
For CSV I would use the StringField() procedure.

[Axolotl]

use the search to find a lot of string parsing examples here on the forum.
Try different searchs (because sometimes the search is very selective.....)

Very good example codes found in here.
SplitString to list or array with option double-quotes (CSV)