Lines of a ListView Gadget to ClipBoard

[fvillanova]

Hello everyone, today I'm asking for help again.
All programs I currently code in PureBasic are extremely fast and compact, but there is still a small and simple routine for copying TXT lines from a "Listview gadget" which is very... but very slow by PureBasic standards.
I'm going to put a small example here, so someone can help me do it faster, see:

Code: Select all

DisableDebugger
Enumeration
  #Window_0
  #Button_0
  #Button_1
  #Button_2
  #Listview_0
  #Text_1 
  #Text_2
EndEnumeration
    OpenWindow(#Window_0, 380, 150, 771, 536, "Putting strings in and out of a ListView Gadget",  #PB_Window_SystemMenu | #PB_Window_MinimizeGadget | #PB_Window_TitleBar | #PB_Window_ScreenCentered )
    ButtonGadget(#Button_0, 700, 500, 60, 25, "E X I T")
    ButtonGadget(#Button_1, 100, 100, 120, 25, "Putting IN")
    ButtonGadget(#Button_2, 100, 130, 120, 25, "OUT to ClipBoard")
    TextGadget(#Text_1, 200, 104, 200, 20, "",#PB_Text_Right)
    TextGadget(#Text_2, 200, 134, 200, 20, "",#PB_Text_Right)
    ListViewGadget(#Listview_0, 10, 240, 750, 250)
 Global.i i,j,Quit,limit,t1,t2
 Global.s aux,lf
 limit=10000
 lf=Chr(13)+Chr(10)
Repeat
   EventID = WaitWindowEvent()
   If EventID = #PB_Event_CloseWindow  
      Quit = 1
   EndIf   
   If EventID = #PB_Event_Gadget
      Select EventGadget() 
            Case #Button_0
              Quit=1
            Case #Button_1      
              t1 = ElapsedMilliseconds()
                 ClearGadgetItems(#Listview_0)
                 For i=1 To limit: AddGadgetItem(#Listview_0,-1,"PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic"): Next 
              t2 = ElapsedMilliseconds() 
              While WindowEvent(): Wend: SetGadgetText(#Text_1,Str(t2-t1)+" ms")
            Case #Button_2      
              t1 = ElapsedMilliseconds()
                 aux="": j=CountGadgetItems(#Listview_0)-1: For i=0 To j: aux+GetGadgetItemText(#Listview_0,i)+lf: Next
                 SetClipboardText(aux)
              t2 = ElapsedMilliseconds()
              SetGadgetText(#Text_2,Str((t2-t1)/1000)+" s")
      EndSelect    
   EndIf                  
Until Quit = 1

With my computer the times are 1087 ms and 15 seconds, this is an eternity to copy 10000 lines to ClipBoard.
Any ideas to speed up these processes?

[BarryG]

When adding to, editing, or deleting from a ListIconGadget, you can turn off visual updates to make it faster.

For reading the ListIconGadget lines, you're suffering the long-standing issue of slow string building in PureBasic:

Code: Select all

For i=0 To j: aux+GetGadgetItemText(#Listview_0,i)+lf: Next

It's quicker to save the ListIconGadget data to a file and then just read the file back into a string variable (crazy, I know).

Here's my example with both of these approaches done. I've added "<" to the end of each ListIcon line to show where it ends. You can omit that if necessary, or use another unique character/string and then ReplaceString() it later.

Code: Select all

DisableDebugger

Enumeration
  #Window_0
  #Button_0
  #Button_1
  #Button_2
  #Listview_0
  #Text_1 
  #Text_2
EndEnumeration

OpenWindow(#Window_0, 380, 150, 771, 536, "Putting strings in and out of a ListView Gadget",  #PB_Window_SystemMenu | #PB_Window_MinimizeGadget | #PB_Window_TitleBar | #PB_Window_ScreenCentered )
ButtonGadget(#Button_0, 700, 500, 60, 25, "E X I T")
ButtonGadget(#Button_1, 100, 100, 120, 25, "Putting IN")
ButtonGadget(#Button_2, 100, 130, 120, 25, "OUT to ClipBoard")
TextGadget(#Text_1, 200, 104, 200, 20, "",#PB_Text_Right)
TextGadget(#Text_2, 200, 134, 200, 20, "",#PB_Text_Right)
ListViewGadget(#Listview_0, 10, 240, 750, 250)

Global.i i,j,Quit,limit,t1,t2
Global.s aux,lf

limit=10000
lf=Chr(13)+Chr(10)

Repeat
  EventID = WaitWindowEvent()
  If EventID = #PB_Event_CloseWindow  
    Quit = 1
  EndIf   
  If EventID = #PB_Event_Gadget
    Select EventGadget() 
      Case #Button_0
        Quit=1
      Case #Button_1      
        t1 = ElapsedMilliseconds()
        ClearGadgetItems(#Listview_0)
        SendMessage_(GadgetID(#Listview_0),#WM_SETREDRAW,0,0) ; Turn off visual updates when adding, deleting or editing items.
        AddGadgetItem(#Listview_0,-1,"start<")
        For i=1 To limit
          AddGadgetItem(#Listview_0,-1,"PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic<")
        Next 
        AddGadgetItem(#Listview_0,-1,"end<")
        t2 = ElapsedMilliseconds()
        SendMessage_(GadgetID(#Listview_0),#WM_SETREDRAW,1,0) ; Turn visual updates back on.
        While WindowEvent(): Wend: SetGadgetText(#Text_1,Str(t2-t1)+" ms")
      Case #Button_2      
        t1 = ElapsedMilliseconds()
        j=CountGadgetItems(#Listview_0)-1
        f$=GetTemporaryDirectory()+"fast-string-build"
        If CreateFile(0,f$)
          For i=0 To j
            WriteString(0,GetGadgetItemText(#Listview_0,i))
          Next
          CloseFile(0)
          If ReadFile(0,f$)
            aux=ReadString(0,#PB_File_IgnoreEOL)
            CloseFile(0)
          EndIf
          DeleteFile(f$)
        EndIf
        SetClipboardText(aux)
        t2 = ElapsedMilliseconds()
        SetGadgetText(#Text_2,Str((t2-t1)/1000)+" s")
    EndSelect
  EndIf
  
Until Quit = 1

[RASHAD]

Hi

Code: Select all

           Case #Button_2      
              t1 = ElapsedMilliseconds()
                Text$ =""
                For item = 0 To CountGadgetItems(#Listview_0)
                  Text$ = Text$ + GetGadgetItemText(#Listview_0,item) + #CRLF$
                Next
                ClearClipboard()
                SetClipboardText(Text$)
              t2 = ElapsedMilliseconds()
              SetGadgetText(#Text_2,Str((t2-t1)/1000)+" s")

[BarryG]

That's what he's already doing, Rashad (see what I quoted of his code). It's PureBasic's slow string building that is the bottleneck. Fred really needs to finally address this, once and for all.

[Demivec]

Here's a different approach. It incorporates BarryG's solution for speeding of the addition of items to the ListIconGadget but uses a different method to export them to the clipboard.

The method addresses the slowdowns with the string concatenation. Each time a GadgetItem is added to the string for the clipboard it increases it's length. In subsequent additions the new length of the string has to be traversed to find the ending and this naturally gets longer and longer with each addition.

To address this a smaller collection of strings is concatenated, 250 in this case instead of the total 10000. This is then added to the string for the clipboard only a total of 40 times (40x250 = 10000). By making this small change the time reduces from 7 sec to 158 ms. When added together the number of smaller strings should be limited so as to not create a length longer than around 34000 characters (according to testing). Because all of the line items are about the same length this works out to be 250 of them at a time.

I've included a sample of loops that are setup to use a variable step rate if you want to determine the grouping on the fly by sampling a few of the items to see how many to group together based on your needs.

Code: Select all

Enumeration
  #Window_0
  #Button_0
  #Button_1
  #Button_2
  #Listview_0
  #Text_1 
  #Text_2
EndEnumeration
OpenWindow(#Window_0, 380, 150, 771, 536, "Putting strings in and out of a ListView Gadget",  #PB_Window_SystemMenu | #PB_Window_MinimizeGadget | #PB_Window_TitleBar | #PB_Window_ScreenCentered )
ButtonGadget(#Button_0, 700, 500, 60, 25, "E X I T")
ButtonGadget(#Button_1, 100, 100, 120, 25, "Putting IN")
ButtonGadget(#Button_2, 100, 130, 120, 25, "OUT to ClipBoard")
TextGadget(#Text_1, 200, 104, 200, 20, "",#PB_Text_Right)
TextGadget(#Text_2, 200, 134, 200, 20, "",#PB_Text_Right)
ListViewGadget(#Listview_0, 10, 240, 750, 250)
Global.i i,j, j2, k, Quit,limit,t1,t2
Global.s aux, aux2, lf
CallDebugger
limit=10000
lf=Chr(13)+Chr(10)
Repeat
  EventID = WaitWindowEvent()
  If EventID = #PB_Event_CloseWindow  
    Quit = 1
  EndIf   
  If EventID = #PB_Event_Gadget
    Select EventGadget() 
      Case #Button_0
        Quit=1
      Case #Button_1      
        t1 = ElapsedMilliseconds()
        SendMessage_(GadgetID(#Listview_0),#WM_SETREDRAW,0,0) ; Turn off visual updates when adding, deleting or editing items.
        ClearGadgetItems(#Listview_0)
        For i=1 To limit: AddGadgetItem(#Listview_0,-1,"" + i + ": PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic"): Next 
        SendMessage_(GadgetID(#Listview_0),#WM_SETREDRAW,1,0) ; Turn visual updates back on.
        t2 = ElapsedMilliseconds() 
        While WindowEvent(): Wend: SetGadgetText(#Text_1,Str(t2-t1)+" ms")
      Case #Button_2   
        t1 = ElapsedMilliseconds()
        aux="": j=CountGadgetItems(#Listview_0)-1
        ;Break string additions into smaller portions by accumlating (aux2) only a certain number of
        ;sub strings (GadgetitemsText()) and then adding the sub string to the total string (aux).
        ;Needs the additional variables j2, k and aux2.s defined.
        For i=0 To j Step 250
          j2 = i + 249 ;step -1
          If j2 > j: j2 = j: EndIf 
          aux2 = ""
          For k = i To j2
            aux2+GetGadgetItemText(#Listview_0,k)+lf
          Next
          Debug Len(aux2)
          aux + aux2
        Next
        
        ;For reference, here is an alternate form of the previous 2 loops that allows a variable step (st).
        ;Needs the additional variables st, j2, k and aux2.s defined.
        ; st = 250 ;step size
        ; i = 0
        ; While i < j 
        ;   j2 = i + st - 1 ;step -1
        ;   If j2 > j: j2 = j: EndIf 
        ;   aux2 = ""
        ;   For k = i To j2
        ;     aux2+GetGadgetItemText(#Listview_0,k)+lf
        ;   Next
        ;   aux + aux2
        ;   i + st
        ; Wend
        SetClipboardText(aux)
        t2 = ElapsedMilliseconds()
        SetGadgetText(#Text_2,Str((t2-t1)) + " ms")
    EndSelect    
  EndIf                  
Until Quit = 1

[fvillanova]

WOW! There's nothing like asking someone who understands.
I have many programs that need to take strings from the ListView and pass them to the clipboard, now it will be instantaneous!
Both solutions are fantastic, thanks to Demivec and BarryG.
I have no words to thank you, it will help me a lot.
thank you again

[chi]

Using a "virtual" ListIconGadget instead of the ListViewGadget and a faster string concatenation:

Code: Select all

CompilerIf #PB_Compiler_Debugger
  CompilerError "Disable the Debugger..."
CompilerEndIf

#LVSICF_NOINVALIDATEALL = 1
#LVSICF_NOSCROLL = 2
#LVN_ODCACHEHINT = #LVN_FIRST - 13

#ItemCount = 10000

Global Dim Item.s(#ItemCount)

;- Faster string concatenation: Alternative method (by Karig) https://www.purebasic.fr/english/viewtopic.php?t=69411

Structure CatBuffer
  capacity.i
  length.i
  text.s
EndStructure

Procedure EmptyCatBuffer(*cat.CatBuffer)
  Protected blank$ = ""
  *cat\capacity = 0
  *cat\length = 0
  PokeS(@*cat\text, blank$, 0)
EndProcedure

Procedure AddToCatBuffer(*cat.CatBuffer, string$)
  ; Appends string$ to the buffer. If the buffer is too small, this procedure
  ; will double the buffer's size before appending the string. Simply doubling
  ; the size on overflow uses more memory but necessitates fewer reallocations
  ; of memory as the buffer grows.
  
  Protected new_length = *cat\length + StringByteLength(string$)
  Protected new_capacity, old_text$, *address
  
  If new_length > *cat\capacity
    new_capacity = *cat\capacity
    If new_capacity = 0
      new_capacity = 1024 * 4
    EndIf
    
    While new_length > new_capacity
      new_capacity * 2
    Wend
    
    old_text$ = *cat\text
    *cat\text = Space(new_capacity)
    *cat\capacity = new_capacity
    PokeS(@*cat\text, old_text$)
  EndIf
  
  *address = @*cat\text + *cat\length
  PokeS(*address, string$)
  *cat\length = new_length
EndProcedure

;-

Enumeration
  #Window_0
  #Button_0
  #Button_1
  #Button_2
  #ListIcon_0
  #Text_1
  #Text_2
EndEnumeration

Procedure WinCallback(hwnd, msg, wParam, lParam)
  result = #PB_ProcessPureBasicEvents
  Select msg
    Case #WM_NOTIFY
      *nmh.NMHDR = lParam
      Select *nmh\code
        Case #LVN_ODCACHEHINT
          result = 0
        Case #LVN_GETDISPINFO
          *nmlvd.NMLVDISPINFO = lParam
          If *nmlvd\item\mask & #LVIF_TEXT
            *nmlvd\item\pszText = @Item(*nmlvd\item\iItem)
          EndIf
      EndSelect
  EndSelect
  ProcedureReturn result
EndProcedure

OpenWindow(#Window_0, 380, 150, 771, 536, "Putting strings in and out of a ListIcon Gadget", #PB_Window_SystemMenu|#PB_Window_MinimizeGadget|#PB_Window_TitleBar|#PB_Window_ScreenCentered)
SetWindowCallback(@WinCallback())
ButtonGadget(#Button_0, 700, 500, 60, 25, "E X I T")
ButtonGadget(#Button_1, 100, 100, 120, 25, "Putting IN")
ButtonGadget(#Button_2, 100, 130, 120, 25, "OUT to ClipBoard")
TextGadget(#Text_1, 230, 104, 200, 20, "", #PB_Text_Right)
TextGadget(#Text_2, 230, 134, 200, 20, "", #PB_Text_Right)
ListIconGadget(#ListIcon_0, 10, 240, 750, 250, "", 725, #LVS_OWNERDATA|#LVS_NOCOLUMNHEADER)

Repeat
  event = WaitWindowEvent()
  Select event
    Case #PB_Event_Gadget
      Select EventGadget()
          
        Case #Button_0
          event = #PB_Event_CloseWindow
          
        Case #Button_1
          SetGadgetText(#Text_1, "clearing view...")
          SendMessage_(GadgetID(#ListIcon_0), #LVM_SETITEMCOUNT, 0, #LVSICF_NOINVALIDATEALL|#LVSICF_NOSCROLL)
          t1 = ElapsedMilliseconds()
          SendMessage_(GadgetID(#ListIcon_0), #LVM_SETITEMCOUNT, #ItemCount, #LVSICF_NOINVALIDATEALL|#LVSICF_NOSCROLL)
          For i=0 To #ItemCount-1
            Item(i) = Str(i) + " PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic" + #CRLF$
          Next
          t2 = ElapsedMilliseconds()
          SetGadgetText(#Text_1, Str(t2 - t1) + " ms")
          
        Case #Button_2
          SetGadgetText(#Text_2, "setting clipboard...")
          t1 = ElapsedMilliseconds()
          For i=0 To #ItemCount-1
            AddToCatBuffer(cat.CatBuffer, Item(i))
          Next
          SetClipboardText(cat\text)
          t2 = ElapsedMilliseconds()
          SetGadgetText(#Text_2, Str(t2 - t1) + " ms")
          EmptyCatBuffer(cat)
          
      EndSelect
  EndSelect
Until event = #PB_Event_CloseWindow

[fvillanova]

Using a "virtual" ListIconGadget instead of the ListViewGadget and a faster string concatenation:

Code: Select all

CompilerIf #PB_Compiler_Debugger
  CompilerError "Disable the Debugger..."
CompilerEndIf

#LVSICF_NOINVALIDATEALL = 1
#LVSICF_NOSCROLL = 2
#LVN_ODCACHEHINT = #LVN_FIRST - 13

#ItemCount = 10000

Global Dim Item.s(#ItemCount)

;- Faster string concatenation: Alternative method (by Karig) https://www.purebasic.fr/english/viewtopic.php?t=69411

Structure CatBuffer
  capacity.i
  length.i
  text.s
EndStructure

Procedure EmptyCatBuffer(*cat.CatBuffer)
  Protected blank$ = ""
  *cat\capacity = 0
  *cat\length = 0
  PokeS(@*cat\text, blank$, 0)
EndProcedure

Procedure AddToCatBuffer(*cat.CatBuffer, string$)
  ; Appends string$ to the buffer. If the buffer is too small, this procedure
  ; will double the buffer's size before appending the string. Simply doubling
  ; the size on overflow uses more memory but necessitates fewer reallocations
  ; of memory as the buffer grows.
  
  Protected new_length = *cat\length + StringByteLength(string$)
  Protected new_capacity, old_text$, *address
  
  If new_length > *cat\capacity
    new_capacity = *cat\capacity
    If new_capacity = 0
      new_capacity = 1024 * 4
    EndIf
    
    While new_length > new_capacity
      new_capacity * 2
    Wend
    
    old_text$ = *cat\text
    *cat\text = Space(new_capacity)
    *cat\capacity = new_capacity
    PokeS(@*cat\text, old_text$)
  EndIf
  
  *address = @*cat\text + *cat\length
  PokeS(*address, string$)
  *cat\length = new_length
EndProcedure

;-

Enumeration
  #Window_0
  #Button_0
  #Button_1
  #Button_2
  #ListIcon_0
  #Text_1
  #Text_2
EndEnumeration

Procedure WinCallback(hwnd, msg, wParam, lParam)
  result = #PB_ProcessPureBasicEvents
  Select msg
    Case #WM_NOTIFY
      *nmh.NMHDR = lParam
      Select *nmh\code
        Case #LVN_ODCACHEHINT
          result = 0
        Case #LVN_GETDISPINFO
          *nmlvd.NMLVDISPINFO = lParam
          If *nmlvd\item\mask & #LVIF_TEXT
            *nmlvd\item\pszText = @Item(*nmlvd\item\iItem)
          EndIf
      EndSelect
  EndSelect
  ProcedureReturn result
EndProcedure

OpenWindow(#Window_0, 380, 150, 771, 536, "Putting strings in and out of a ListIcon Gadget", #PB_Window_SystemMenu|#PB_Window_MinimizeGadget|#PB_Window_TitleBar|#PB_Window_ScreenCentered)
SetWindowCallback(@WinCallback())
ButtonGadget(#Button_0, 700, 500, 60, 25, "E X I T")
ButtonGadget(#Button_1, 100, 100, 120, 25, "Putting IN")
ButtonGadget(#Button_2, 100, 130, 120, 25, "OUT to ClipBoard")
TextGadget(#Text_1, 230, 104, 200, 20, "", #PB_Text_Right)
TextGadget(#Text_2, 230, 134, 200, 20, "", #PB_Text_Right)
ListIconGadget(#ListIcon_0, 10, 240, 750, 250, "", 725, #LVS_OWNERDATA|#LVS_NOCOLUMNHEADER)

Repeat
  event = WaitWindowEvent()
  Select event
    Case #PB_Event_Gadget
      Select EventGadget()
          
        Case #Button_0
          event = #PB_Event_CloseWindow
          
        Case #Button_1
          SetGadgetText(#Text_1, "clearing view...")
          SendMessage_(GadgetID(#ListIcon_0), #LVM_SETITEMCOUNT, 0, #LVSICF_NOINVALIDATEALL|#LVSICF_NOSCROLL)
          t1 = ElapsedMilliseconds()
          SendMessage_(GadgetID(#ListIcon_0), #LVM_SETITEMCOUNT, #ItemCount, #LVSICF_NOINVALIDATEALL|#LVSICF_NOSCROLL)
          For i=0 To #ItemCount-1
            Item(i) = Str(i) + " PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic PureBasic" + #CRLF$
          Next
          t2 = ElapsedMilliseconds()
          SetGadgetText(#Text_1, Str(t2 - t1) + " ms")
          
        Case #Button_2
          SetGadgetText(#Text_2, "setting clipboard...")
          t1 = ElapsedMilliseconds()
          For i=0 To #ItemCount-1
            AddToCatBuffer(cat.CatBuffer, Item(i))
          Next
          SetClipboardText(cat\text)
          t2 = ElapsedMilliseconds()
          SetGadgetText(#Text_2, Str(t2 - t1) + " ms")
          EmptyCatBuffer(cat)
          
      EndSelect
  EndSelect
Until event = #PB_Event_CloseWindow

Hi Chi, sorry I only saw your solution now, I had to travel and I didn't read the messages here on the forum, it was a complicated few days with a lot of work.
I see that it is the fastest of all, I will use it when I need to copy thousands of lines to the clipboard (> 10000), it will help a lot within my programs.
Thank you very much for sharing and teaching how to create this routine.
Villanova

[BarryG]

Each time a GadgetItem is added to the string for the clipboard it increases it's length. In subsequent additions the new length of the string has to be traversed to find the ending and this naturally gets longer and longer with each addition.

I still have hope that Fred will address this string-building issue one day so we don't have to use workarounds.