Going Loopy

[Evil1]

Hi all, just needing some pointers on the correct way or best way to do this :-

The grabfile() procedure is essentially readfile(0,file$) : while not eof(0) : Text$ = ReadString(0) : addgadgetitem(item, -1, text$) : Wend

Using this approach causes the Main Window to become unresponsive and this is understandable, but I can't figure out the best way to do this without making the Window unresponsive.

I wanted to create a progressbar and I tried using a Thread - I got it working but the compiled executable was at least 4 times slower than it was when I ran the application in the IDE. (I believe this has something to do with Threadsafe and the slow down of strings)

Anyway, just really looking for some input on the best way to do this e.g. are threads the way to go or is there a better way to handle the loading of a file into a gadget?

Code: Select all

If CreateMenu(0, WindowID(Main)) 
      MenuTitle("File")
      MenuItem(#Menu_File_Open, "&Open..." + #TAB$ + "Ctrl+O")
      AddKeyboardShortcut(Main, #PB_Shortcut_Control|#PB_Shortcut_O, #Menu_File_Open)
      MenuBar()
      MenuItem(#Menu_File_Exit, "&Exit" + #TAB$ + "Ctrl+Q")
      AddKeyboardShortcut(Main, #PB_Shortcut_Control|#PB_Shortcut_Q, #Menu_File_Exit)
    EndIf
          
    BindEvent(#PB_Event_Menu, @OnMenuEvent(),Main)
    BindEvent(#PB_Event_Gadget, @OnEventGadget())
    BindEvent(#PB_Event_SizeWindow, @OnEventSize(),Main)
    BindEvent(#PB_Event_CloseWindow, @Quit())
    ;BindEvent(#PB_Event_Timer,@OnTimer())
      
Repeat: WaitWindowEvent() : ForEver

Code: Select all

Procedure OnMenuEvent()
  Select EventMenu()
    Case #Menu_File_Open
      ;Progress()
      filepicker()
      grabfile(file$)
    Case #Menu_File_Exit
      Quit()
    EndSelect
EndProcedure

[skywalk]

You have to show your threaded code.
The performance hit for threadsafe compilation is no way near 4x
Non-threaded or single thread apps are rare. All my apps are threaded.

[Marc56us]

If the gadget to load is an EditorGadget then just use the #PB_File_IgnoreEOL flag in ReadString() to speed up the loading considerably.

Code: Select all

OpenFile(0, "")
While Not Eof(0)
     Txt$ = ReadString(0, #PB_File_IgnoreEOL)
Wend
CloseFile(0)

SetGadgetText(0, Txt$)

You won't be able to use ProgressBar, but anyway the user won't have time to see it, even with several thousands of lines.

If the gadget is a list, menu or other then this should not be a problem either, as this type of gadget is not designed to load masses of data as it is also very slow for the end user.
It is better to use a linked list or array and make a partial display on gadget.
Here it is a question of human/machine interface design (Read the guides to effective GUI design)

Tip: Internal calculations to display a progress bar will also slow down (sometimes considerably) the work of the rest of the program. A trick to let the user know that something is still going on is to make a progress bar that loops (e.g. 25%, 50%, 75%, 100%, then again 25%, 50%...) with timer 1/2 sec, so the user is busy and sees less waiting time.

[Evil1]

@Skywalk I will get some code up tomorrow to show you what I was doing

@Marc56us, several thousand - lol, I am iterating through several million!

It's taking about 5 seconds to go through a 100MB file, but I have files up to 6GB in size!!!

[skywalk]

Also, provide a file link or dummy code generator to make your big file for testing purposes.
It's most likely you are running into the dreaded string concatenation problem.
That is slow whether threaded or not!
Instead of x$ = x$ + new$, you use memory buffer and CopyMemoryString()...

[IdeasVacuum]

... and what OS?

If loading a List, in Windows you can hold the display update until the List is complete - that makes a difference.

[mrv2k]

If you disable the list gadget things will speed up considerably. I can load 40,000 lines plus almost instantly with the following code...

Code: Select all

SendMessage_(GadgetID(#YourList),#WM_SETREDRAW,#False,0) 
readfile(0,file$)
while not eof(0)
   Text$ = ReadString(0)
   addgadgetitem(#YourList, -1, text$)
Wend
SendMessage_(GadgetID(#YourList),#WM_SETREDRAW,#True,0) 
RedrawWindow_(GadgetID(#YourList), #Null, #Null, #RDW_ERASE | #RDW_FRAME | #RDW_INVALIDATE | #RDW_ALLCHILDREN)

This is windows only.

[Marc56us]

It's taking about 5 seconds to go through a 100MB file, but I have files up to 6GB in size!!!

6GB: forget it, even many good editor can't read such file. (Already numbering the lines, they often cannot exceed 999,999)

If you want to work in editor mode with big file, you can use the Scintilla gadget.
One of my application with Scintilla processes a 700 000 lines file (23Mb) (loaded in 3 sec from a SSD disk on my i7)

In general, when processing large files, we do not load them in memory or as a gadget, we work in flow mode (for text) or by block (for images).

We've been doing this for decades, especially in Perl with files much larger than RAM, it's not a problem.

PS. Sources of large text files for testing: %windir%\*.log (simply concatenate several of them. ie: COPY txt1 + txt2 + txt3 Fulltxt.txt), webserver log (access.log), /var/log in unix etc

[Evil1]

If you disable the list gadget things will speed up considerably. I can load 40,000 lines plus almost instantly with the following code...

Code: Select all

SendMessage_(GadgetID(#YourList),#WM_SETREDRAW,#False,0) 
readfile(0,file$)
while not eof(0)
   Text$ = ReadString(0)
   addgadgetitem(#YourList, -1, text$)
Wend
SendMessage_(GadgetID(#YourList),#WM_SETREDRAW,#True,0) 
RedrawWindow_(GadgetID(#YourList), #Null, #Null, #RDW_ERASE | #RDW_FRAME | #RDW_INVALIDATE | #RDW_ALLCHILDREN)

This is windows only.

I was using hidegadget(editor,1) and hidegadget(editor,0) - I assume the API is the fastest solution?

It's taking about 5 seconds to go through a 100MB file, but I have files up to 6GB in size!!!

6GB: forget it, even many good editor can't read such file. (Already numbering the lines, they often cannot exceed 999,999)

If you want to work in editor mode with big file, you can use the Scintilla gadget.
One of my application with Scintilla processes a 700 000 lines file (23Mb) (loaded in 3 sec from a SSD disk on my i7)

In general, when processing large files, we do not load them in memory or as a gadget, we work in flow mode (for text) or by block (for images).

We've been doing this for decades, especially in Perl with files much larger than RAM, it's not a problem.

PS. Sources of large text files for testing: %windir%\*.log (simply concatenate several of them. ie: COPY txt1 + txt2 + txt3 Fulltxt.txt), webserver log (access.log), /var/log in unix etc

I am merely scannning through the files and pulling bits out, so even though I may be scanning a file that is 6GB in size I may only be pulling out 1000 lines or so.


In regards to the thread :-

I created a procedure called Progress that creates the thread grabfile()

Code: Select all

Procedure Progress(x = 0, y = 0, width = 300, height = 110)
   ;ltim=ElapsedMilliseconds()
  Progress = OpenWindow(#PB_Any, x, y, width, height, "", #PB_Window_SystemMenu)
  ProgressBar_0 = ProgressBarGadget(#PB_Any, 20, 10, 250, 20, 0, 100)
  Cancel = ButtonGadget(#PB_Any, 90, 70, 100, 25, "Cancel")
  tim = TextGadget(#PB_Any, 100, 50, 150, 20,"")
  CreateThread(@grabfile(),0)
   
  ;If ElapsedMilliseconds()-starttime >=1:starttime.q=ElapsedMilliseconds():SetGadgetState(ProgressBar_0,pg):EndIf
  ;If ElapsedMilliseconds()-starttime >=1:starttime.q=ElapsedMilliseconds():SetGadgetText(tim,Str(ElapsedMilliseconds()/1000-ltim/1000)+" Second(s)"):EndIf
  Repeat : event = WaitWindowEvent() : Until pg>=100
  CloseWindow(progress)
EndProcedure

But I had to disable the Bindevent on #PB_Event_Menu as the progress procedure could not create a thread in a binded event callback :-

Code: Select all

;BindEvent(#PB_Event_Menu, @OnMenuEvent(),Main)
    BindEvent(#PB_Event_Gadget, @OnEventGadget())
    BindEvent(#PB_Event_SizeWindow, @OnEventSize(),Main)
    BindEvent(#PB_Event_CloseWindow, @Quit())
    ;BindEvent(#PB_Event_Timer,@OnTimer())
    
    Repeat 
    event=WaitWindowEvent()
    Select Event
      Case #PB_Event_Menu
        Select EventGadget()
           Case #Menu_File_Open
              filepicker()
              progress()
              ;grabfile(file$)
            
            Case #Menu_File_Exit
              Quit()
          EndSelect
      EndSelect
    ForEver

Then in my grabfile(t) procedure I do

Code: Select all

Repeat
  If ReadFile(0,file$)
    While Not Eof(0)
       Text$ = ReadString(0)
        pg = 100 * Loc(0) / length 
          If ElapsedMilliseconds()-starttime >=20:starttime.q=ElapsedMilliseconds():SetGadgetState(ProgressBar_0,pg):EndIf
          If ElapsedMilliseconds()-starttime >=20:starttime.q=ElapsedMilliseconds():SetGadgetText(tim,Str(ElapsedMilliseconds()/1000-ltim/1000)+" Second(s)"):EndIf
       if Findstring(text$,"Error")
         addgadgetitem(Editor,-1,text$)
      endif
    Wend
 Endif
Until pg>=100

But that runs really slow when compiled (on 100MB text file), but takes about 5 seconds when run from the IDE, I am sure it is my understanding (lack of) threads that is the issue

[Marc56us]

If I understand correctly, the goal is to search for lines containing a keyword (here "Error") and add the line to a list ?

Running this part of the program as a Thread is not going to be any faster, it just allows not to block the rest of the program, i.e. the user will still be able to interrupt the program, move the window etc.

[Evil1]

If I understand correctly, the goal is to search for lines containing a keyword (here "Error") and add the line to a list ?

Running this part of the program as a Thread is not going to be any faster, it just allows not to block the rest of the program, i.e. the user will still be able to interrupt the program, move the window etc.

Yes that is right, I am not attempting to speed up the program by using a thread - I am merely trying to use it to show some loading progress, which is sort of working :-

https://ibb.co/Wtvm82x

[Marc56us]

Okay, so I overdid it a little bit, but then you'll see what we can do with the threads.
(doesn't follow the awful KillThread, don't do it like that )

A progress bar slows down the management of the rest of the program considerably, another solution is to update it only from time to time.
i = counter of line
Example: update only every 1000 lines

Code: Select all

        If i / 1000 = i % 1000
            StatusBarProgress(0, 0, i, #PB_StatusBar_Raised, 0, Lines)
        EndIf

Full tested sample.

Code: Select all

EnableExplicit

CompilerIf #PB_Compiler_Thread = #False
    MessageRequester("Warning !!",
                     "You must enable ThreadSafe support in compiler options",
                     #PB_MessageRequester_Ok )
    End
CompilerEndIf 

; XIncludeFile "Search_Error.pbf"
; Copy/past from form designer to do one single file for forum
; ------------------------------------------------------------------------------------
;
; This code is automatically generated by the FormDesigner.
; Manual modification is possible to adjust existing commands, but anything else will be dropped when the code is compiled.
; Event procedures needs to be put in another source file.
;

Enumeration FormWindow
  #Window_0
EndEnumeration

Enumeration FormGadget
  #Lst_Errors
  #Btn_Open
  #Str_Search
  #Btn_Search
  #Btn_Stop
  #Btn_Quit
EndEnumeration

Enumeration FormFont
  #Font_Window_0_0
EndEnumeration

LoadFont(#Font_Window_0_0,"Consolas", 10)

Declare ResizeGadgetsWindow_0()


Procedure OpenWindow_0(x = 0, y = 0, width = 600, height = 400)
  OpenWindow(#Window_0, x, y, width, height, "Show Errors", #PB_Window_SystemMenu | #PB_Window_MinimizeGadget | #PB_Window_MaximizeGadget | #PB_Window_SizeGadget | #PB_Window_ScreenCentered)
  CreateStatusBar(0, WindowID(#Window_0))
  AddStatusBarField(200)
  StatusBarProgress(0, 0, 0)
  AddStatusBarField(#PB_Ignore)
  StatusBarText(0, 1, "Ready")
  ListIconGadget(#Lst_Errors, 10, 10, 580, 330, "Line", 60)
  AddGadgetColumn(#Lst_Errors, 1, "Text", 500)
  SetGadgetFont(#Lst_Errors, FontID(#Font_Window_0_0))
  ButtonGadget(#Btn_Open, 10, 345, 100, 25, "Open...")
  StringGadget(#Str_Search, 240, 345, 100, 25, "Error")
  ButtonGadget(#Btn_Search, 135, 345, 100, 25, "Search for")
  DisableGadget(#Btn_Search, 1)
  ButtonGadget(#Btn_Stop, 345, 345, 100, 25, "Stop !")
  ButtonGadget(#Btn_Quit, 490, 345, 100, 25, "Quit")
EndProcedure

Procedure ResizeGadgetsWindow_0()
  Protected FormWindowWidth, FormWindowHeight
  FormWindowWidth = WindowWidth(#Window_0)
  FormWindowHeight = WindowHeight(#Window_0)
  ResizeGadget(#Lst_Errors, 10, 10, FormWindowWidth - 20, FormWindowHeight - StatusBarHeight(0) - 47)
  ResizeGadget(#Btn_Open, 10, FormWindowHeight - 55, 100, 25)
  ResizeGadget(#Str_Search, 240, FormWindowHeight - 55, 100, 25)
  ResizeGadget(#Btn_Search, 135, FormWindowHeight - 55, 100, 25)
  ResizeGadget(#Btn_Stop, 345, FormWindowHeight - 55, 100, 25)
  ResizeGadget(#Btn_Quit, FormWindowWidth - 110, FormWindowHeight - 55, 100, 25)
EndProcedure


; ------------------------------------------------------------------------------------


Global FileName$
Global Nb_Lines.l
Global Txt2Find$
Global ID_Thread

Declare Search_Txt(*Value)

Procedure Show_Infos()
    Nb_Lines = FileSize(FileName$)
    StatusBarText(0, 1, FormatNumber(Nb_Lines, 0, " ", " ") + " bytes")    
EndProcedure

; Here we go...
OpenWindow_0()

; If a filename is provide as parameter or drop on icon, use it directly
If CountProgramParameters() = 1
    FileName$ = ProgramParameter(0)
    Show_Infos()
    SetWindowTitle(#Window_0, FileName$)
    Txt2Find$ = GetGadgetText(#Str_Search)
    If Txt2Find$ <> ""
        ID_Thread = CreateThread(@Search_Txt(), 1)
    EndIf
EndIf


; Thread for searching
Procedure Search_Txt(*Value)
    DisableGadget(#Btn_Stop, 0)
    StatusBarText(0, 1, "Searching...")
    Protected hFile
    Protected i
    Protected Tmp_Line$
    Protected Lines
    ReadFile(hFile, FileName$)  
    ; Count line (EOL)
    While Not Eof(hFile)
        ReadString(hFile)
        Lines + 1
    Wend 
    CloseFile(hFile)
    
    ; Update 20 x per file whatever filesize
    Protected Factor = Lines / 20
    ReadFile(hFile, FileName$)
    While Not Eof(hFile)
        i + 1
        If i / Factor = i % Factor
            StatusBarText(0, 1, Str(i) + " / " + Lines)
            StatusBarProgress(0, 0, i, #PB_StatusBar_Raised, 0, Lines)
        EndIf
        Tmp_Line$ = ReadString(hFile)
        If FindString(Tmp_Line$, Txt2Find$) > 0 
            AddGadgetItem(#Lst_Errors, -1, Str(i) + Chr(10) + Tmp_Line$)
            Tmp_Line$ = ""
        EndIf
    Wend    
    CloseFile(hFile)
    StatusBarProgress(0, 0, i)
    StatusBarText(0, 1, "OK, Search ended")
EndProcedure


Repeat
    Select WaitWindowEvent()
            
        Case #PB_Event_CloseWindow
            ; Thread will be killed when program quit, 
            ; so whatever If still running...
            End
            
        Case #PB_Event_SizeWindow
            ; Auto generated by form designer 
            ; when change at list on lock gadget position
            ResizeGadgetsWindow_0()
            
        Case #PB_Event_Gadget
            Select EventGadget()
                    
                Case #Btn_Open
                    FileName$ = OpenFileRequester("Log file to analyse", "", "", 0)
                    If FileName$ <> ""
                        DisableGadget(#Btn_Search, 0)
                        Txt2Find$ = GetGadgetText(#Str_Search)
                        Show_Infos()
                    EndIf
                    
                Case #Btn_Search
                    ClearGadgetItems(#Lst_Errors)
                    ID_Thread = CreateThread(@Search_Txt(), 1)
                    
                Case #Btn_Stop
                    ; Bad way, but sometime needed
                    If IsThread(ID_Thread)
                        KillThread(ID_Thread)
                        StatusBarText(0, 1, "Arglll, you have killed my thread |-/")
                    EndIf
                    
                Case #Btn_Quit
                    End
            EndSelect
            
    EndSelect
    
ForEver

End

[/size]
Tested with a file with 700 000 lines on my i7: 6 sec

Edit 1730: actual procedure take care of number of line and update 20 time whatever number of line

Enjoy

[Evil1]

That's great Marc56us, thanks for all your help. This is really helpful

[mk-soft]

Don't forget...
That not work with Linux or maOS

[Marc56us]

Hum, bug

Add "Txt2Find$..." in Procedure

Code: Select all

Procedure Search_Txt(*Value)
    Txt2Find$ = GetGadgetText(#Str_Search)

or in main loop

Code: Select all

	                Case #Btn_Search
                    ClearGadgetItems(#Lst_Errors)
                    Txt2Find$ = GetGadgetText(#Str_Search)
                    ID_Thread = CreateThread(@Search_Txt(), 1)

Otherwise the program will always look for the same term (the one given at the opening).

Don't forget...
That not work with Linux or maOS

I don't have mac and rarely code PB on Linux. What part is not compatible ?

Edit 1800: Test on Linux (slackware) + XFCE + PB 5.61 x64: it work