Cleaning a txt file from dupes ...

[thyr0x1ne]

Im looking for a way to clean a txt file from all the dupes inside

For example transform :
../file.txt
amero
ameri
amerp
amero
amerp
amera

in ../file.txt
ameri
amera

so removing amerp / amero

heres a part of code i wrote :

Code: Select all

Procedure CleanText()
 
 totalstring.l=CountString(textdata,".") + 1  
 ReadFile(1,"file.txt")
 
 While Eof(1) = 0
  text$=text$+ReadString()
 
  For K = 1 To totalstring.l
  
  If FindString(text$,StringField(textdata, K, "."),1) = 1 : Break : EndIf
  If FindString(text$,StringField(textdata, K, "."),1) = 0  
   UseFile(1)
   WriteStringN(StringField(textdata, K, ".")) 
  EndIf
   
  Next K

 Wend
EndProcedure

i was hoping to add a textdata ( parsed by "." as strings ) in a file.txt ONLY if FindString return 0 ( meaning the textdata not yet on file.txt ) , but for now all is written , even dupe words ...

have someone a hint ( lib or routine ) to clean a text file like this , or stop the write process of a string if a line in this txt allready contains the specified string ?

thks for any help

[wilbert]

I would break up the text into an array of words, sort the array and walk through the sorted array. If you compare each word of the sorted array with the previous one and the next one and both don't match, you have a unique word that you can write into a new textfile.

[thyr0x1ne]

damn :/ i cant find the right way ... i used Array as you said , but for large files the exe i did freeze like hell :/

i need a good sleep i think , will lose more neurons tomorrow for now

btw thks for your reply

[Beach]

I'm assuming your data is separated by periods since your code has it listed as such. Take a look at the following code and see if it could work for you.

Code: Select all

;FileHnd = ReadFile(#PB_Any,"file.txt")
;While Eof(FileHnd) = #False : fileData$ + ReadString() : Wend
;CloseFile(FileHnd)

fileData$ = "amero.ameri."
fileData$ + "amerp.amero."
fileData$ + "amerp.amera"

TotalItems = CountString(fileData$,".")
Dim ItemData.s(TotalItems)

For i = 1 To TotalItems+1
  ItemData(i-1) = StringField(fileData$,i,".")
Next

dup.b

For i = 0 To TotalItems
  CurrentItem$ = ItemData(i)
  For x = 0 To TotalItems
    If CurrentItem$ = ItemData(x) And x <> i
      dup = #True
    EndIf
  Next x  
  
  If dup = #False
    newdata$ = newdata$ + CurrentItem$ + "."
  EndIf
  
  dup = #False
Next i

Debug newdata$

EDIT:
After reading your post again I now see that you want to prevent the item from being written in the text file if it already exists in the file. I'm not sure how big your file is - but I would think this method would not be very efficient. I would consider using a database for this. SQLite would work very well for this situation.

[thyr0x1ne]

I decided to make a procedure removing dupe lines *after* the file creation ; here's a working code but of course there must be some limitations ( size of text file certainly ...)

Code: Select all

Procedure RemDupes()

 OpenFile(1,"range.dic") 

 While Eof(1) = 0
   text$=text$+Trim(ReadString())+Chr(42)
 Wend  
   
 textstring.l=CountString(text$,Chr(42))+1
 
 For i=1 To textstring.l
  a$=StringField(text$, i, Chr(42))
  If CountString(text$,a$) > 0  
    clean$=clean$+a$+Chr(42)
    text$=RemoveString(text$,a$+Chr(42))
  EndIf  
 Next
 
 cleanstring.l=CountString(clean$,Chr(42))+1
 
 CloseFile(1)
 
 CopyFile("range.dic", "range.dic.BAK") 
 DeleteFile("range.dic")
 OpenFile(1,"range.dic")
 
 For j=1 To cleanstring.l
  If Len(StringField(clean$, j, Chr(42))) > 0
   WriteStringN(StringField(clean$, j, Chr(42)))
  EndIf 
 Next
 
 CloseFile(1)

EndProcedure

RemDupes()

End

By the way ty for your hint , i saved it to another one of my millions part of source codes