Trouble with: Procedure RemoveListDuplicates()

[millie78526]

TIA !
Trouble with: Procedure RemoveListDuplicates()
Sorry having BrainFog :

Code: Select all

;EnableExplicit

Global NewList WordsList.s() , NewList Index_List.s() 

; Directly add words to the list using AddElement
AddElement(WordsList())
WordsList() = "Apple"
AddElement(WordsList())
WordsList() = "apple"
AddElement(WordsList())
WordsList() = "Apple"
AddElement(WordsList())
WordsList() = "Banana"
AddElement(WordsList())
WordsList() = "Banana"
AddElement(WordsList())
WordsList() = "banana"
AddElement(WordsList())
WordsList() = "apple"  ; Intentional case-insensitive duplicate
AddElement(WordsList())
WordsList() = "Carrot"
AddElement(WordsList())
WordsList() = "Banana"  ; Exact duplicate

; Sort the list to facilitate duplicate removal
SortList(WordsList(), #PB_Sort_Ascending | #PB_Sort_NoCase)

; Function to remove duplicates from a sorted list
Procedure RemoveListDuplicates()
  If ListSize(WordsList()) > 0  ; Only process if there is at least one element
    Define previousLine.s = "zzzzzzzz"
    ForEach WordsList()
      If previousLine = WordsList()
        DeleteElement(WordsList(), ListIndex(WordsList()))  ; Remove the duplicate element
        NextElement(WordsList()) 
      EndIf 
      If previousLine <> WordsList()  ; Update previous line
        Debug "WordsList() = " + ListSize(WordsList())
        Debug "ListSize() = " + ListSize(Index_List())
        AddElement(Index_List())
        Index_List() = WordsList() 
      EndIf
    Next
  EndIf
EndProcedure

; Remove duplicates
RemoveListDuplicates()

; Function to print the list
Procedure PrintWordsList()
  ForEach WordsList()
    Debug "Word: " + WordsList()
  Next
  ForEach Index_List()
    Debug "Word: " + Index_List()
  Next
  
EndProcedure

; Print the words in the list
PrintWordsList()

; Clean up
FreeList(WordsList())
MessageRequester("End of Program", "The program has completed execution.")

[pjay]

I would use a map for this, you wouldn't need to sort the list either:

Code: Select all

Procedure RemoveListDuplicates(casesensitive = #True)
  Protected lcword.s : NewMap st.i()
  If ListSize(WordsList()) > 0  ; Only process if there is at least one element
    ForEach WordsList()
      lcword = WordsList() : If casesensitive = #False : lcword = LCase(lcword) : EndIf
      If st(lcword) = 0 : st() = 1 : Else : DeleteElement(WordsList()) : EndIf
    Next
  EndIf
  FreeMap(st())
EndProcedure

[AZJIO]

Code: Select all

Procedure RemoveListDuplicates(CaseSensitive = #True)
	Protected w$
	Protected NewMap Unique()
	ForEach WordsList()
		If CaseSensitive
			If FindMapElement(Unique(), WordsList())
				DeleteElement(WordsList())
			Else
				AddMapElement(Unique(), WordsList())
			EndIf
		Else
			w$ = LCase(WordsList())
			If FindMapElement(Unique(), w$)
				DeleteElement(WordsList())
			Else
				AddMapElement(Unique(), w$)
			EndIf
		EndIf
	Next
EndProcedure

[millie78526]

Thank you pjay :
My code is part of a much larger program that Reads in .txt files , often over 5k lines of text ,
and then breaks lines into words , then removes Duplicates .
Then creates a ListView of Unique words ,
User can then click on Resort_Button to Resort Words according to CheckBox Ascending or Descending .
Could you explain what's going on here? :

Code: Select all

      If st(lcword) = 0  
        st() = 1  

Thanks

[AZJIO]

https://www.purebasic.fr/english/viewto ... 30#p608430
SplitMapByWords

[infratec]

Code: Select all

Procedure RemoveListDuplicates()
  If ListSize(WordsList()) > 0  ; Only process if there is at least one element
    Define previousLine.s = "zzzzzzzz"
    ForEach WordsList()
      If LCase(previousLine) = LCase(WordsList())
        DeleteElement(WordsList())  ; Remove the duplicate element
        ;NextElement(WordsList()) 
      EndIf 
      If previousLine <> WordsList()  ; Update previous line
        Debug "WordsList() = " + ListSize(WordsList())
        Debug "ListSize() = " + ListSize(Index_List())
        AddElement(Index_List())
        Index_List() = WordsList() 
      EndIf
      previousLine = WordsList()
    Next
  EndIf

EndProcedure

[Demivec]

Could you explain what's going on here? :

Code: Select all

      If st(lcword) = 0  
        st() = 1  

St() is a map that is indexed by strings and contains integers. If the map element at index lcword equals zero then it is set equal to one. This condition is only met when the map element is first created. As a result when the same map element is referenced again its value is now 1 and this signals a duplicate word and so the current word entry is deleted from WordsList().

[millie78526]

Thanks Guys ,
You have given me a lot to work with .
Thanks for the lesson on Map .
I have never worked with Map before .
Thanks again .