Text File Insert Append Prepend Replace Delete line

[IdeasVacuum]

Expanding on this post: http://www.purebasic.fr/english/viewtop ... =12&t=3626 about inserting a line into a text file, below is a procedure for each of Insert, Append, Prepend, Replace and Delete a whole line of text.

First thing to note is that they all work and should be cross-platform Second thing to note is that they could all be improved/replaced to work faster with large files

Contents of very simple test file "C:\TestFile.txt" (so short that of course the speed is good):

Test Line 1
Test Line 22
Test Line 333
Test Line 4444
Test Line 55555
Test Line 666666
Test Line 7777777
Test Line 88888888
Test Line 999999999
Test Line 01010101010
Test Line 1111111111111
Test Line 1212121212121212

Code: Select all

Procedure DeleteLine(iFileID,sFileName.s,iDelLineNum.i)
;------------------------------------------------------
; ***** Fast code by Bernd (infratec), made fail friendly by IdeasVacuum *****

        iReturnVal.i = #True
             iSize.l = FileSize(sFileName)
          iReadLen.l = 0
                 i.i = 1
                 n.i = 0
           iLength.l = 0

        If iSize > 0

                 If ReadFile(iFileID,sFileName)

                             *Buffer = AllocateMemory(iSize)
                          If *Buffer

                                     iReadLen = ReadData(iFileID, *Buffer, iSize)
                                                CloseFile(iFileID)

                                  If(iReadLen = iSize)

                                         While i < iDelLineNum

                                               If PeekA(*Buffer + n) = $0A
                                                      i + 1
                                               EndIf

                                               n + 1
                                         Wend

                                         *Dest = *Buffer + n
                                             n = 0

                                         While PeekA(*Dest + n) <> $0A
                                               n + 1
                                         Wend

                                         *Source = *Dest + n + 1
                                         iLength = (*Buffer + iSize) - *Source

                                         MoveMemory(*Source, *Dest, iLength)

                                         iSize - n - 1

                                         If CreateFile(iFileID, sFileName)

                                                  WriteData(iFileID, *Buffer, iSize)
                                                  CloseFile(iFileID)
                                         Else
                                                  MessageRequester("File Issue","File create failed",#PB_MessageRequester_Ok)
                                                  iReturnVal = #False
                                         EndIf

                                  Else
                                         MessageRequester("Memory Issue","File read failed",#PB_MessageRequester_Ok)
                                         iReturnVal = #False
                                  EndIf

                                  FreeMemory(*Buffer)
                          Else
                                  MessageRequester("Memory Issue","Memory allocation failed",#PB_MessageRequester_Ok)
                                  iReturnVal = #False
                          EndIf
                 Else
                          MessageRequester("File Issue","File open failed",#PB_MessageRequester_Ok)
                          iReturnVal = #False
                 EndIf
        Else
                 MessageRequester("File Issue","File empty",#PB_MessageRequester_Ok)
                 iReturnVal = #False
        EndIf

        ProcedureReturn(iReturnVal)

EndProcedure


Procedure InsertLine(iFileID,sFileName.s,iLine.i,sNewLine.s,iMode.i)
;-------------------------------------------------------------------
          NewList sData.s()
          iReturnVal = #TRUE

          If ReadFile(iFileID,sFileName)

                          While Eof(iFileID) = 0

                                   AddElement(sData())
                                   sData() = ReadString(iFileID)

                          Wend

                          CloseFile(iFileID)

                  SelectElement(sData(),iLine)
                  InsertElement(sData())
                                sData() = sNewLine

                  If CreateFile(iFileID,sFileName)

                          FirstElement(sData())

                          For i = 1 To ListSize(sData())

                                   WriteStringN(iFileID,sData(),iMode)
                                    NextElement(sData())

                          Next

                          CloseFile(iFileID)
                Else
                        MessageRequester("File Issue","File create failed",#PB_MessageRequester_Ok)
                        iReturnVal = #FALSE
                EndIf

        Else
                MessageRequester("File Issue","File read failed",#PB_MessageRequester_Ok)
                iReturnVal = #FALSE

        EndIf

        FreeList(sData())

        ProcedureReturn(iReturnVal)

EndProcedure


Procedure PrependLine(iFileID,sFileName.s,sPrependLine.s,iMode.i)
;----------------------------------------------------------------

          iFileLen.l = 0
          iReadLen.l = 0
           iStrLen.l = 0
          iReturnVal = #TRUE

          If ReadFile(iFileID,sFileName)

                          iFileLen = Lof(iFileID)
                       If(iFileLen > 0)

                                  *MemFile = AllocateMemory(iFileLen)
                               If *MemFile

                                     iReadLen = ReadData(iFileID,*MemFile,iFileLen)

                               EndIf

                               CloseFile(iFileID)

                               If(iFileLen = iReadLen)

                                      If CreateFile(iFileID,sFileName)

                                              WriteStringN(iFileID,sPrependLine,iMode)
                                                 WriteData(iFileID,*MemFile,iFileLen)
                                                 CloseFile(iFileID)
                                      Else
                                             MessageRequester("File Issue","File create failed",#PB_MessageRequester_Ok)
                                             iReturnVal = #FALSE
                                      EndIf

                               Else
                                      MessageRequester("Memory Issue","File read failed",#PB_MessageRequester_Ok)
                                      iReturnVal = #FALSE
                               EndIf
                       Else
                               MessageRequester("File Issue","File empty",#PB_MessageRequester_Ok)
                               iReturnVal = #FALSE
                       EndIf

          Else
                       MessageRequester("File Issue","File read failed",#PB_MessageRequester_Ok)
                       iReturnVal = #FALSE

          EndIf

          If *MemFile : FreeMemory(*MemFile) : EndIf

          ProcedureReturn(iReturnVal)

EndProcedure


Procedure AppendLine(iFileID,sFileName.s,sAppendLine.s,iMode.i)
;--------------------------------------------------------------

           iStrLen.l = 0
          iFileLen.l = 0
          iReturnVal = #TRUE

               If OpenFile(iFileID,sFileName)

                          iFileLen = Lof(iFileID)
                       If(iFileLen > 0)

                               FileSeek(iFileID,iFileLen)

                               ; line to be appended
                               WriteStringN(iFileID,sAppendLine,iMode)

                               CloseFile(iFileID)

                       Else
                               MessageRequester("File Issue","File empty",#PB_MessageRequester_Ok)
                               iReturnVal = #FALSE

                       EndIf

              Else
                       MessageRequester("File Issue","Open file failed",#PB_MessageRequester_Ok)
                       iReturnVal = #FALSE
              EndIf

          ProcedureReturn(iReturnVal)

EndProcedure


Procedure ReplaceLine(iFileID,sFileName.s,iRepLineNum.i,sReplacement.s,iMode.i)
;-----------------------------------------------------------------------------
          NewList sData.s()
          iReturnVal = #TRUE


          If ReadFile(iFileID,sFileName)

                          While Eof(iFileID) = 0

                                   AddElement(sData())
                                   sData() = ReadString(iFileID)

                          Wend

                          CloseFile(iFileID)

                  SelectElement(sData(),(iRepLineNum - 1)) ; -1 because first element number is zero
                                sData() = sReplacement

                  If CreateFile(iFileID,sFileName)

                            FirstElement(sData())

                            For i = 1 To ListSize(sData())

                                     WriteStringN(iFileID,sData(),iMode)
                                      NextElement(sData())

                            Next

                            CloseFile(iFileID)
                  Else
                          MessageRequester("File Issue","File create failed",#PB_MessageRequester_Ok)
                          iReturnVal = #FALSE
                  EndIf

          Else
                  MessageRequester("File Issue","File read failed",#PB_MessageRequester_Ok)
                  iReturnVal = #FALSE

          EndIf

          FreeList(sData())

          ProcedureReturn(iReturnVal)

EndProcedure

    iReturn.i = 0

    ;### Test: replace line 5 ###
    iReturn = ReplaceLine(0,"C:\TestFile.txt",5,"Replacement Text",iMode.i)

    ;### Test: insert line after line 8 ###
    iReturn = InsertLine(0,"C:\TestFile.txt",8,"Inserted Line",#PB_Ascii)

    ;### Test: delete line 3 ###
    iReturn = DeleteLine(0,"C:\TestFile.txt",3)

    ;### Test: append a line ###
    iReturn = AppendLine(0,"C:\TestFile.txt","Appended Line",#PB_Ascii)

    ;### Test: Prepend string ###
    iReturn = PrependLine(0,"C:\TestFile.txt","Prepended Line",#PB_Ascii)

    End

So, the challenge is to improve the speed, but without any API or other dependencies.......

Edit 01: Free Memory; Use WriteStringN
Edit 02: File Fail friendly
Edit03: DeleteLine() replaced with faster code by Bernd

[infratec]

Hi,

my first look over it shows something terrible:

You don't free your allocated memories


And it is more cross platform if you use WriteStringN(),
because textfiles in Linux uses not #CRLF$ as line termination.
WriteStringN() should handle this automatically.

If I have time I try to improve some stuff.

Bernd

[infratec]

Hi,

maybe an improvement:

Code: Select all

Procedure AppendLine(iFileID, sAppendLine.s, iMode.i = 0, sFileName.s = "")
;--------------------------------------------------------------

  If iFileID = #PB_Any
    FileID = OpenFile(#PB_Any, sFileName)
    If Not FileID
      MessageRequester("File Issue", "Open file failed", #PB_MessageRequester_Ok)
      ProcedureReturn #False
    EndIf
  Else
    FileID = iFileID
  EndIf
      
  FileSeek(FileID, Lof(FileID))
  WriteStringN(FileID, sAppendLine, iMode)
  
  If iFileID = #PB_Any
    CloseFile(FileID)
  EndIf
  
  ProcedureReturn #True
  
EndProcedure

AppendLine(#PB_Any, "Appended Line", #PB_Ascii, "TestFile.txt")

Why an improvement?

1. The file can already be opened
2. If the file not exists, the text is appended
3. WriteStringN() used.
4. A return value added
5. Optional parameters

Don't know if it is useful.

Bernd

[IdeasVacuum]

Hi Bernd

You are absolutely correct, WriteStringN is a much better choice in this case. Memory allocated should be freed yes, as these procedures should 'stand alone'.

Ditching the opening of the file is not good, the procedures then are not stand alone and if the file is mission critical, it should be saved immediately.

Having a return value to confirm that the process worked is a good idea.

#PB_Any does not belong here in my view, the app could be handling several files, so FileID specific is important.

What we really want to do though is improve the speed.......

[infratec]

Here is a part of what you really want

Code: Select all

Procedure DeleteLine(iFileID,sFileName.s,iDelLineNum.i,iMode.i)
;-------------------------------------------------------------
          NewList sData.s()
          iReturnVal = #True

          If ReadFile(iFileID,sFileName)

                          While Eof(iFileID) = 0

                                   AddElement(sData())
                                   sData() = ReadString(iFileID)

                          Wend

                          CloseFile(iFileID)

                  SelectElement(sData(),(iDelLineNum - 1)) ; -1 because first element number is zero
                  DeleteElement(sData())

                  If CreateFile(iFileID,sFileName)

                          FirstElement(sData())

                          For i = 1 To ListSize(sData())

                                   WriteStringN(iFileID,sData(),iMode)
                                    NextElement(sData())

                          Next

                          CloseFile(iFileID)
                  Else
                          MessageRequester("File Issue","File create failed",#PB_MessageRequester_Ok)
                          iReturnVal = #False
                  EndIf

          Else
                  MessageRequester("File Issue","File read failed",#PB_MessageRequester_Ok)
                  iReturnVal = #False

          EndIf

          FreeList(sData())

          ProcedureReturn(iReturnVal)

EndProcedure
        

Procedure DeleteLineBKK(iFileID,sFileName.s,iDelLineNum.i,iMode.i)
;-------------------------------------------------------------
  iReturnVal = #True
  
  
  Size = FileSize(sFileName)
  
  If Size > 0
    
    If ReadFile(iFileID,sFileName)
      *Buffer = AllocateMemory(Size)
      If *Buffer
        
        If ReadData(iFileID, *Buffer, Size) = Size
          CloseFile(iFileID)
          
          i = 1
          n = 0
          While i < iDelLineNum
            If PeekA(*Buffer + n) = $0A
              i + 1
            EndIf
            n + 1
          Wend
          *Dest = *Buffer + n
 ;Debug PeekS(*Dest, 100)
          n = 0
          While PeekA(*Dest + n) <> $0A
            n + 1
          Wend
          *Source = *Dest + n + 1
 ;Debug PeekS(*Source, 100)
 
          Length = (*Buffer + Size) - *Source

          MoveMemory(*Source, *Dest, Length)
 
          Size - n - 1

          CreateFile(iFileID, sFileName)
          WriteData(iFileID, *Buffer, Size)
          
        EndIf
        
        FreeMemory(*Buffer)
        
      EndIf
      CloseFile(iFileID)
    EndIf
  EndIf
      
  ProcedureReturn(iReturnVal)

EndProcedure




Procedure CreateTxtFile(Filename$)
  File = CreateFile(#PB_Any, Filename$)
  If File
    For i = 1 To 1000000
      WriteStringN(File, LSet(Str(i), 8) + Space(70))
    Next i
    CloseFile(File)
  EndIf
EndProcedure




CompilerIf #True

CreateTxtFile("test.txt")
StartTime = ElapsedMilliseconds()
DeleteLine(0, "test.txt", 100, 0)
EndTime = ElapsedMilliseconds()
MessageRequester("info", "Time: " + Str(EndTime - StartTime))

CompilerEndIf


CreateTxtFile("test.txt")
StartTime = ElapsedMilliseconds()
DeleteLineBKK(0, "test.txt", 100, 0)
EndTime = ElapsedMilliseconds()
MessageRequester("info", "Time: " + Str(EndTime - StartTime))

I think my version is over 2 times faster

Bernd

Edit: New file was 1 byte to large. Sorry.

[infratec]

Hi,

additonal 6% faster:

Code: Select all

Procedure DeleteLineBKK2(iFileID,sFileName.s,iDelLineNum.i,iMode.i)
;-------------------------------------------------------------
  iReturnVal = #False
  
  Size = FileSize(sFileName)
  
  If Size > 0
    
    If ReadFile(iFileID,sFileName)
      *Buffer = AllocateMemory(Size)
      If *Buffer
        
        If ReadData(iFileID, *Buffer, Size) = Size
          CloseFile(iFileID)
          
          CreateFile(iFileID, sFileName)
          
          i = 1 : n = 0
          While i < iDelLineNum
            If PeekA(*Buffer + n) = $0A : i + 1 : EndIf
            n + 1
          Wend
          
          WriteData(iFileID, *Buffer, n - 1)
                    
;Debug PeekS(*Buffer + n, 100)
          i = n
          While PeekA(*Buffer + i) <> $0A
            i + 1
          Wend
;Debug PeekS(*Buffer + n, 100)
          
          WriteData(iFileID, *Buffer + i, Size - i)         
 
          iReturnVal = #True
          
        EndIf
        
        FreeMemory(*Buffer)
        
      EndIf
      CloseFile(iFileID)
    EndIf
  EndIf
      
  ProcedureReturn(iReturnVal)

EndProcedure

I don't use MoveMemory() anymore
I also use now the return value.

If I have more sparetime I will look into the other procedures.

Bernd

[infratec]

Ditching the opening of the file is not good, the procedures then are not stand alone and if the file is mission critical, it should be saved immediately.

#PB_Any does not belong here in my view, the app could be handling several files, so FileID specific is important.

Hm,

if you don't want this feature, than use #PB_Any.
Than it open and close the file as you want it.

When you use a real FileID, it uses the already opened file.
Maybe someone else want it this way.

So you are not limited, but it also offers the other way.

(Only my thoughts)

Bernd

[infratec]

I have more time than I expected

Here is my version for InsertLine()

Code: Select all

Procedure InsertLineBKK(iFileID, sFileName.s, iLine.i, sNewLine.s, iMode.i)
;-------------------------------------------------------------
  iReturnVal = #False
  
  Size = FileSize(sFileName)
  
  If Size > 0
    
    If ReadFile(iFileID,sFileName)
      *Buffer = AllocateMemory(Size)
      If *Buffer
        
        If ReadData(iFileID, *Buffer, Size) = Size
          CloseFile(iFileID)
          
          CreateFile(iFileID, sFileName)
          
          i = 1 : n = 0
          While i < iLine
 CompilerIf #PB_Compiler_OS = #PB_OS_MacOS
            If PeekA(*Buffer + n) = $0D
 CompilerElse
            If PeekA(*Buffer + n) = $0A
 CompilerEndIf
              i + 1
            EndIf
            n + 1
          Wend
          
          WriteData(iFileID, *Buffer, n)
          WriteStringN(iFileID, sNewLine, iMode)
          WriteData(iFileID, *Buffer + n, Size - n)         
 
          iReturnVal = #True
          
        EndIf
        
        FreeMemory(*Buffer)
        
      EndIf
      CloseFile(iFileID)
    EndIf
  EndIf
      
  ProcedureReturn(iReturnVal)

EndProcedure

Also more than 2 times faster.

I added a CompilerIf, so now it should also work on a Mac.
(I think in MS$ it is CRLF in Linux it is LF and in Mac it is a CR)

But now I have todo something else.

Bernd

[IdeasVacuum]

Hi Bernd

Wow, that was weird, I posted an update based on your DeleteLine() (which is not 2 but over 3 times faster on my PC), then low and behold, find that you have submitted more, faster code! I'll play with it tomorrow, looks like you have solved the two main speed issues.

[infratec]

Hi,

first, I found a small Bug in my DeleteLine() procedure.
Here is the corrected (and cross-compatible) version:

Code: Select all

Procedure DeleteLine(iFileID,sFileName.s,iDelLineNum.i,iMode.i)
;-------------------------------------------------------------
  iReturnVal = #False
  
  Size = FileSize(sFileName)
  
  If Size > 0
    
    If ReadFile(iFileID,sFileName)
      *Buffer = AllocateMemory(Size)
      If *Buffer
        
        If ReadData(iFileID, *Buffer, Size) = Size
          CloseFile(iFileID)
          
          CreateFile(iFileID, sFileName)
          
          i = 1 : n = 0
          While i < iDelLineNum
CompilerIf #PB_Compiler_OS = #PB_OS_MacOS
            If PeekA(*Buffer + n) = $0D : i + 1 : EndIf
CompilerElse
            If PeekA(*Buffer + n) = $0A : i + 1 : EndIf
CompilerEndIf
            n + 1
          Wend
;Debug PeekS(*Buffer + n, 100)
          
          WriteData(iFileID, *Buffer, n)
          
          i = n
 CompilerIf #PB_Compiler_OS = #PB_OS_MacOS
          While PeekA(*Buffer + i) <> $0D
CompilerElse
          While PeekA(*Buffer + i) <> $0A
CompilerEndIf
            i + 1
          Wend
          i + 1
;Debug PeekS(*Buffer + i, 100)
          
          WriteData(iFileID, *Buffer + i, Size - i)         
          
          iReturnVal = #True
          
        EndIf
        
        FreeMemory(*Buffer)
        
      EndIf
      CloseFile(iFileID)
    EndIf
  EndIf
      
  ProcedureReturn(iReturnVal)

EndProcedure

And now my last contribution:

Code: Select all

Procedure ReplaceLine(iFileID, sFileName.s, iRepLineNum.i, sReplacement.s, iMode.i)
;-------------------------------------------------------------
  iReturnVal = #False
  
  Size = FileSize(sFileName)
  
  If Size > 0
    
    If ReadFile(iFileID,sFileName)
      *Buffer = AllocateMemory(Size)
      If *Buffer
        
        If ReadData(iFileID, *Buffer, Size) = Size
          CloseFile(iFileID)
          
          CreateFile(iFileID, sFileName)
          
          i = 1 : n = 0
          While i < iRepLineNum
CompilerIf #PB_Compiler_OS = #PB_OS_MacOS
            If PeekA(*Buffer + n) = $0D : i + 1 : EndIf
CompilerElse
            If PeekA(*Buffer + n) = $0A : i + 1 : EndIf
CompilerEndIf
            n + 1
          Wend
;Debug PeekS(*Buffer + n, 100)
          
          WriteData(iFileID, *Buffer, n)
          WriteStringN(iFileID, sReplacement, iMode)
                    
          i = n
CompilerIf #PB_Compiler_OS = #PB_OS_MacOS
          While PeekA(*Buffer + i) <> $0D
CompilerElse
          While PeekA(*Buffer + i) <> $0A
CompilerEndIf
            i + 1
          Wend
          i + 1
;Debug PeekS(*Buffer + i, 100)
          
          WriteData(iFileID, *Buffer + i, Size - i)         
 
          iReturnVal = #True
          
        EndIf
        
        FreeMemory(*Buffer)
        
      EndIf
      CloseFile(iFileID)
    EndIf
  EndIf
      
  ProcedureReturn(iReturnVal)

EndProcedure

Best regards,

Bernd

[Demivec]

infratec's code can be further simplified by turning the sets of alternate compiler statements such as:

Code: Select all

CompilerIf #PB_Compiler_OS = #PB_OS_MacOS
            If PeekA(*Buffer + n) = $0D : i + 1 : EndIf
CompilerElse
            If PeekA(*Buffer + n) = $0A : i + 1 : EndIf
CompilerEndIf

into a single set defining a constant and then just using that constant in each of the affected lines:

Code: Select all

CompilerIf #PB_Compiler_OS = #PB_OS_MacOS
  #eol = $0D
CompilerElse
  #eol = $0A
CompilerEndIf 

            If PeekA(*Buffer + n) = #eol : i + 1 : EndIf

[IdeasVacuum]

Considering these procedures are for text file handling, I think it would be more cross-platform to detect possible EOL's rather than compile for a specific platform - after all, you may well be using Windows and need to edit a MAC or Linux OS file.

Would something like this work in the real world?

Code: Select all

If( (PeekA(*Buffer + n) = $0A) Or (PeekA(*Buffer + n) = $0D) ) : i + 1 : EndIf

[Demivec]

Considering these procedures are for text file handling, I think it would be more cross-platform to detect possible EOL's rather than compile for a specific platform - after all, you may well be using Windows and need to edit a MAC or Linux OS file.

Would something like this work in the real world?

Code: Select all

If( (PeekA(*Buffer + n) = $0A) Or (PeekA(*Buffer + n) = $0D) ) : i + 1 : EndIf

No. Because Windows uses CRLF a windows file should detect two lines but for MAC or Linux OS it should detect only 1 line. This determination could be made by using a runtime parameter or examining the file to determine its format.

[IdeasVacuum]

examining the file to determine its format

Yes, I think that's the best approach.