AddLine for Text Files

[BackupUser]

Restored from previous forum. Originally posted by Franco.

This is code for adding a line in text files.
Like the other Procedures (GetLine,SetLine,MaxLine) it works only if you open a file with ReadFile(), before you use this command. Maybe some day (if Fred will change OpenFile) it will work with OpenFile. At the end of the procedure the file is reopened with ReadFile().

Code: Select all

; (c) 2001 - Franco's template - absolutely freeware
; Adds a line to a text file
; return value 1 = new line added
; return value 0 = function failed
Procedure AddLine(File,FileName$,Line,String$)
  UseFile(File)
  LineNumber=1
  If Line>1
    While  LineNumber0 
      Size=SizeORG-Location
      *Buffer=AllocateMemoryBank(1,Size,0)
      FileSeek(Location) 
      ReadData(*Buffer,Size)
      CloseFile(0)
      If OpenFile(0, FileName$)
        UseFile(0)
        FileSeek(Location)
        WriteStringN(String$)
        WriteData(*Buffer,Size)
        CloseFile(0)
        If ReadFile(0, FileName$) And SizeORG<Lof()
          ProcedureReturn 1
        Else
          ProcedureReturn 0
        EndIf
      EndIf
    Else
      ExitAddLineWhile:
      ProcedureReturn 0
    EndIf
  Else
    ProcedureReturn 0
  EndIf
EndProcedure

Have a nice day...
Franco

(ReadMemory and WriteMemory renamed to ReadData and WriteData)

Edited by - Franco on 06 December 2001 23:14:29

[IdeasVacuum]

I want to do something like this but I'm not sure how this procedure ever worked. Of course the syntax has changed since 2001, but how does the line

Code: Select all

Size=SizeORG-Location

work when all 3 vars are unassigned? Indeed, not sure about

Code: Select all

While  LineNumber0 

either......

[kenmo]

Hmm. Do you want to insert a line of text (that is, increasing the number of lines) or replace a line of text?

There are a few ways to do both. I started writing an insertion procedure, but realized
(a) you might want a line replace procedure, and
(b) the method of reading/buffering and rewriting the file each time you change a line (like Franco did here) is probably inefficient. Might be better to store and change the text in an array or list, then just write a Flush() procedure that re-builds the text file only when you want.

[IdeasVacuum]

Hmm, I can insert a new string (new line) into the required location, but the next line in the file is overwritten:

Code: Select all

Procedure InsertLine(iFileID,sFileName.s,iLine.i,sNewLine.s,iMode.i)
;-------------------------------------------------------------------
      sLine.s = ""
      iSize.l = 0
iInsertPosn.l = 0
   qFileLen.q = 0
       iLen.i = 0
       iExtra = 4

               If((iMode = #PB_Ascii) Or (iMode = #PB_UTF8)) : iExtra = 2 : EndIf

               If OpenFile(iFileID,sFileName)

                     For i = 1 To iLine

                                    sLine = ReadString(iFileID,iMode)
                                     iLen = StringByteLength(sLine,iMode)
                              iInsertPosn = iInsertPosn + iLen + iExtra
                     Next i

                         FileSeek(iFileID,iInsertPosn)
                     WriteStringN(iFileID,sNewLine,iMode)
                        CloseFile(iFileID)

              EndIf

EndProcedure

Edit: I could of course read the entire file, close it, create it, write including new line, close it. However, seems like a sledge hammer to crack a nut:

Code: Select all

Procedure InsertLine(iFileID,sFileName.s,iLine.i,sNewLine.s,iMode.i)
;-------------------------------------------------------------------
    sHeader.s = ""
      sBody.s = ""
      sLine.s = ""

               If ReadFile(iFileID,sFileName)

                     For i = 1 To iLine

                                      sLine = ReadString(iFileID,iMode)
                                    sHeader = sHeader + sLine + #CRLF$
                     Next i

                     While Eof(iFileID) = 0

                                    sLine = ReadString(iFileID,iMode)
                                    sBody = sBody + sLine + #CRLF$

                     Wend

                     CloseFile(iFileID)

              EndIf

              If CreateFile(iFileID,sFileName)

                      WriteString(iFileID,sHeader,iMode)
                     WriteStringN(iFileID,sNewLine,iMode)
                      WriteString(iFileID,sBody,iMode)
                     CloseFile(iFileID)
              EndIf

EndProcedure

Works well with a small file (approx 40 chars per line, 500 lines)

[IdeasVacuum]

Hi kenmo

Yeah, I want to insert a new line of text. For small files, the hammer-crack-nut method works well and for my current project it is fine. Still, there should be a 'best method' for any text file, big or small.

[flaith]

Hi,

just an idea :
using sqlite in memory import each line of the text file then use the sql command "insert" and then save the new file text ?

[netmaestro]

just an idea :
using sqlite in memory import each line of the text file then use the sql command "insert" and then save the new file text ?

Yes, simple and elegant way to do it without reinventing the wheel. Only one downside: Place the line UseSQLiteDatabase() anywhere in your code and you add 400k to the size of the executable. Not that that should necessarily bother you, I'm just saying. It costs 400k to avoid tackling the logic to insert a line of text where you want it. There's a term for programmers who feel that's a good deal, they're called .Net coders

p.s. No offense, I'm betting you didn't know about the 400k

[kenmo]

Filesize overhead aside, doesn't using an SQL database for this seem like overkill anyway?

You'd have to read through the file, send Insert/Update/Select queries to the database, loop through them to write it out....

Might as well use PB's linked lists. Less overhead, probably slightly faster, no need for queries and string escaping.

Just use SelectElement() indexed by a line number, then you can InsertElement() or just change lines of text as necessary. Saving is also simple.

[IdeasVacuum]

Using Linked List:

Code: Select all

Procedure InsertLine(iFileID,sFileName.s,iLine.i,sNewLine.s,iMode.i)
;-------------------------------------------------------------------
NewList sData.s()

              If ReadFile(iFileID,sFileName)

                      While Eof(iFileID) = 0

                               AddElement(sData())
                               sData() = ReadString(iFileID)

                      Wend

                      CloseFile(iFileID)
              EndIf

              SelectElement(sData(),iLine)
              InsertElement(sData())
                            sData() = sNewLine

              If CreateFile(iFileID,sFileName)

                      FirstElement(sData())

                      For i = 1 To ListSize(sData())

                               WriteStringN(iFileID,sData(),iMode)
                                NextElement(sData())

                      Next

                      CloseFile(iFileID)
              EndIf

              FreeList(sData())

EndProcedure

Edit: The LinkedList method is much faster when processing a large-ish file (40 chars per line x 60,000 lines), but it still takes a while at around 15 seconds.

[kenmo]

What sort of files are you working with? How often will you be writing to them?

When I suggested linked lists I assumed you'd load the file once, change the linked list while you use your program, and eventually save it back to the file.

That would be much faster, if you're making multiple changes to it.

Reading and writing the entire file every time you change one line is probably going to be pretty slow no matter how you do it...
(I would think ReadData and WriteData would be faster than thousands of ReadLines and WriteLines though.)

[flaith]

just an idea :
using sqlite in memory import each line of the text file then use the sql command "insert" and then save the new file text ?

Yes, simple and elegant way to do it without reinventing the wheel. Only one downside: Place the line UseSQLiteDatabase() anywhere in your code and you add 400k to the size of the executable. Not that that should necessarily bother you, I'm just saying. It costs 400k to avoid tackling the logic to insert a line of text where you want it. There's a term for programmers who feel that's a good deal, they're called .Net coders

p.s. No offense, I'm betting you didn't know about the 400k

hehe, from you, there's no offense, i agree i totally forgot the 400kb

[IdeasVacuum]

What sort of files are you working with? How often will you be writing to them?

I am working with relatively small ASCII text files, 500 to 2000 lines. The lines (strings) are around 40 chars but vary in length. Both of the hammer-nut-crack Procedures I have posted are perfectly fine for my app, even though they do need to be saved every time a line is inserted (which is good practise in terms of preserving the data).

However, I'm minded that someone else will one day make the same forum search, so it would be good to have a defacto solution. Read/Write data is probably it, but it would seem to require finding the end of each line (CR, CR/LF), which could be messy.......

[IdeasVacuum]

Here is an attempt using ReadData-WriteData:

Code: Select all

Procedure InsertLine(iFileID,sFileName.s,iLine.i,sNewLine.s,iMode.i)
;-------------------------------------------------------------------

   sTest.s = ""
iFileLen.l = 0
iReadLen.l = 0
 iStrLen.l = 0
 iHdrLen.l = 0

               If ReadFile(iFileID,sFileName)

                          iFileLen = Lof(iFileID)
                          Debug "File Length-->" + Str(iFileLen)
                       If(iFileLen > 0)

                                  *MemFile = AllocateMemory(iFileLen)
                                  *MemCopy = AllocateMemory(iFileLen)
                                  *MemPosn = *MemFile

                               If *MemFile

                                     iReadLen = ReadData(iFileID,*MemFile,iFileLen)

                               EndIf

                               CloseFile(iFileID)

                               If(iFileLen = iReadLen)

                                      ; Collection of lines ('File Header') before insertion position
                                      For i = 1 To iLine

                                                   sTest = PeekS(*MemPosn,-1,iMode)
                                                 iStrLen = StringByteLength(sTest,iMode)
                                                 Debug "Line " + Str(i) + " Length-->" + Str(iStrLen)
                                                *MemPosn = *MemPosn + iStrLen
                                                 iHdrLen = iHdrLen + iStrLen
                                      Next

                                      Debug "Header Length-->" + Str(iHdrLen)

                                      ; line to be inserted
                                          iStrLen = StringByteLength(sNewLine,iMode)
                                      *MemNewLine = AllocateMemory(iStrLen)
                                      PokeS(*MemNewLine,sNewLine,iStrLen,iMode)

                                      If CreateFile(iFileID,sFileName)

                                              WriteData(iFileID,*MemFile,iHdrLen)
                                              WriteData(iFileID,*MemNewLine,iStrLen)
                                              WriteData(iFileID,*MemPosn,(iFileLen - iHdrLen))
                                              CloseFile(iFileID)
                                      EndIf

                               Else
                                      MessageRequester("Memory Issue","File read failed",#PB_MessageRequester_Ok)
                               EndIf
                       Else
                               MessageRequester("File Issue","File empty",#PB_MessageRequester_Ok)

                       EndIf
              EndIf

EndProcedure

;### Test: insert a string @ line 5 ###

    InsertLine(0,"C:\TestFile.txt",5,"Inserted Line",#PB_Ascii)

    End

....it's no good though, the first PeekS collects the entire file!

The test file "C:\TestFile.txt" contents:

Test Line 1111111111
Test Line 222222222222222
Test Line 33333333333333333333
Test Line 4444444444444444444444444
Test Line 555555555555555555555555555555
Test Line 66666666666666666666666666666666666
Test Line 7777777777777777777777777777777777777777
Test Line 888888888888888888888888888888888888888888888
Test Line 99999999999999999999999999999999999999999999999999

[idle]

you could also use a map and list storing the address from delimited words stored in the map
it'd cut down on memory and allow you to manage large text files and do multiple inserts, deletes and replaces.
If you wanted it to be line based you'd need to store line numbers in the list

Code: Select all

Structure fileIndex 
   index.i
   lineNumber.i  
EndStructure 

something like this.

Read in data

Code: Select all

While Not Eof(fn) 
    LineNumber + 1
    line = ReadString(fn) 
    ct = CountString(line,delimiter)
    For a = 1 To ct +1
      word.s = StringField(line,a,delimiter)
      textmap(word)=word 
      AddElement(index())
      index()\index = @textmap(word)
      index()\line = LineNumber
     ;add delimeter to?  
    Next 
  Wend 

insert data

Code: Select all

 
 FirstElement(index())
 For a = 1 To position  
    NextElement(index())
 Next  
       
  LineNumber = index()\line
  ct = CountString(string,delimiter)
  For a = 1 To ct +1
     word.s = StringField(string,a,delimiter)
     textmap(word)=word 
     AddElement(index())
     index()\index = @textmap(word)
     index()\line = LineNumber
     ;add delimiter  
   Next 
  

write it out

Code: Select all

 
 FileSeek(fn,0) 
   ForEach index() 
     word = PeekS(PeekI(index()\index)) 
     WriteString(fn,word)
    Next 
  CloseFile(fn)

[IdeasVacuum]

Hello Idle

I think reading a large file line-by-line is too slow, as per my earlier example using LinkedList.

See also http://www.purebasic.fr/english/viewtop ... 13&t=46407