Recursive thread function?

camille · Post by **camille** » Sun Dec 13, 2020 9:37 am

Hello!

I'm trying to do a threaded function that needs to call itself for recursion...

This is what I have done so far:

Code: Select all


  DeclareModule SimpleScanDir

    Structure SSD_PARAMS
      Map ItemInfos.i()
      path.s
      pattern.s
    EndStructure

    NewMap PathExclusions.s()
    PathExclusions(".") = ""
    PathExclusions("..") = ""

    CompilerIf #PB_Compiler_OS = #PB_OS_Windows
      PathExclusions("$recycle.bin") = ""
      PathExclusions("system volume information") = ""
    CompilerEndIf

    Declare.i ScanDir(*SSDThread.SSD_PARAMS)

  EndDeclareModule


  ; *************************************************************************************************
  ; *************************************************************************************************

  Module SimpleScanDir

    EnableExplicit

    Procedure.i ScanDir(*SSDThread.SSD_PARAMS)
      Protected.i hDir

      If *SSDThread\path = "" : ProcedureReturn 0 : EndIf

      If Right(*SSDThread\path, 1) <> #PS$ : *SSDThread\path + #PS$ : EndIf

      hDir = ExamineDirectory(#PB_Any, *SSDThread\path, *SSDThread\pattern)
      If hDir
        While NextDirectoryEntry(hDir)
          Select DirectoryEntryType(hDir)
            Case #PB_DirectoryEntry_Directory
              ; Skip on defined entries
              If FindMapElement(SimpleScanDir::PathExclusions(), LCase(DirectoryEntryName(hDir)))
                Continue
              EndIf

              ; Recurse
              *SSDThread\path = *SSDThread\path + DirectoryEntryName(hDir) + #PS$
              ScanDir(*SSDThread.SSD_PARAMS)


            Case #PB_DirectoryEntry_File
              Debug "Date: " + Str(DirectoryEntryDate(hDir, #PB_Date_Created)) + " | File: " + *SSDThread\path + DirectoryEntryName(hDir)
              *SSDThread\ItemInfos(*SSDThread\path + DirectoryEntryName(hDir)) = DirectoryEntryDate(hDir, #PB_Date_Created)
          EndSelect
        Wend
        FinishDirectory(hDir)
      EndIf

      ;ClearStructure(*SSDThread, SSD_PARAMS)
      ;FreeMemory(*SSDThread)

      ProcedureReturn MapSize(*SSDThread\ItemInfos())
    EndProcedure

  EndModule

I'm creating the thread like this:

Code: Select all

  *SSDThread.SimpleScanDir::SSD_PARAMS = AllocateMemory(SizeOf(SimpleScanDir::SSD_PARAMS))
  NewMap *SSDThread\ItemInfos()
  *SSDThread\path    = "D:\Install"
  *SSDThread\pattern = "*.*"

  CreateThread(SimpleScanDir::@ScanDir(), *SSDThread)

Debug output:

Code: Select all

Date: 1567678890 | File: D:\Install\Drivers\AMD\AHCI\@Homepage [Download].url
Date: 1567678890 | File: D:\Install\Drivers\AMD\AHCI\Windows 7\v1.2.1.359 [Preferred]\@Supported controllers.txt
Date: 1567678890 | File: D:\Install\Drivers\AMD\AHCI\Windows 7\v1.2.1.359 [Preferred]\x64\amd_sata.cat
Date: 1567678890 | File: D:\Install\Drivers\AMD\AHCI\Windows 7\v1.2.1.359 [Preferred]\x64\amd_sata.inf
Date: 1567678890 | File: D:\Install\Drivers\AMD\AHCI\Windows 7\v1.2.1.359 [Preferred]\x64\amd_sata.sys
Date: 1567678890 | File: D:\Install\Drivers\AMD\AHCI\Windows 7\v1.2.1.359 [Preferred]\x64\amd_xata.sys

Two questions:

01. D:\Install contains about 2560 files, why does the debug output does only list the above?

02. How should I handle the ClearStructure and FreeMemory part? I can't use them directly because of the recursion. Make a new function out of it that is called when the thread has finished?

STARGÅTE · Post by **STARGÅTE** » Sun Dec 13, 2020 10:16 am

I did not check the whole code but this seems not working, if you free a structure and then reading its list size:

Code: Select all

      ClearStructure(*SSDThread, SSD_PARAMS)
      FreeMemory(*SSDThread)

      ProcedureReturn MapSize(*SSDThread\ItemInfos())

camille · Post by **camille** » Sun Dec 13, 2020 10:20 am

Hi STARGÅTE,

yeah, I've seen this and edit the post accordingly. Unfortunately you've seen the old version of my post

STARGÅTE · Post by **STARGÅTE** » Sun Dec 13, 2020 10:32 am

Why you would like to delete the allocated memory inside the function? Don't you use the filled items outside?
You can make a counter inside the structure, which is increased by one on each sub-call and decreased by one on each return.
If the couter is back at 0, you are on the first level and you can delete the structure.

In this like here:

Code: Select all

              *SSDThread\path = *SSDThread\path + DirectoryEntryName(hDir) + #PS$
              ScanDir(*SSDThread.SSD_PARAMS)

You add the sub directory to path, but you never remove it after this call. That means the next call also include the sub-directory from the last directory.

Code: Select all

              
OldPath = *SSDThread\path
*SSDThread\path = *SSDThread\path + DirectoryEntryName(hDir) + #PS$
ScanDir(*SSDThread.SSD_PARAMS)
*SSDThread\path = OldPath

infratec · Post by **infratec** » Sun Dec 13, 2020 10:42 am

path has to be 'local'

Code: Select all

DeclareModule SimpleScanDir
  
  Structure SSD_PARAMS
    Map ItemInfos.i()
    path.s
    pattern.s
  EndStructure
  
  NewMap PathExclusions.s()
  PathExclusions(".") = ""
  PathExclusions("..") = ""
  
  CompilerIf #PB_Compiler_OS = #PB_OS_Windows
    PathExclusions("$recycle.bin") = ""
    PathExclusions("system volume information") = ""
  CompilerEndIf
  
  Declare.i ScanDir(*SSDThread.SSD_PARAMS)
  
EndDeclareModule


; *************************************************************************************************
; *************************************************************************************************

Module SimpleScanDir
  
  EnableExplicit
  
  Procedure ScanDir(*SSDThread.SSD_PARAMS)
    
    Protected hDir.i
    Protected path.s
    
    path = *SSDThread\path
    
    If path <> ""
      
      
      
      If Right(path, 1) <> #PS$
        path + #PS$
      EndIf
      
      hDir = ExamineDirectory(#PB_Any, path, *SSDThread\pattern)
      If hDir
        While NextDirectoryEntry(hDir)
          Select DirectoryEntryType(hDir)
            Case #PB_DirectoryEntry_Directory
              ; Skip on defined entries
              If FindMapElement(SimpleScanDir::PathExclusions(), LCase(DirectoryEntryName(hDir)))
                Continue
              EndIf
              
              ; Recurse
              *SSDThread\path = path + DirectoryEntryName(hDir) + #PS$
              ScanDir(*SSDThread)
              
            Case #PB_DirectoryEntry_File
              Debug "Date: " + Str(DirectoryEntryDate(hDir, #PB_Date_Created)) + " | File: " + path + DirectoryEntryName(hDir)
              *SSDThread\ItemInfos(path + DirectoryEntryName(hDir)) = DirectoryEntryDate(hDir, #PB_Date_Created)
          EndSelect
        Wend
        FinishDirectory(hDir)
      EndIf
    EndIf
    
  EndProcedure
  
EndModule


Define SSDThread.SimpleScanDir::SSD_PARAMS, Thread.i

SSDThread\path    = "c:\tmp"
SSDThread\pattern = "*.*"

Thread = CreateThread(SimpleScanDir::@ScanDir(), @SSDThread)

WaitThread(Thread)

Debug MapSize(SSDThread\ItemInfos())

camille · Post by **camille** » Sun Dec 13, 2020 10:50 am

Why you would like to delete the allocated memory inside the function? Don't you use the filled items outside?

You're absolutely right

This needs to be done in the calling part outside of the module (when I need the thread to be initialized again)

you add the sub directory to path, but you never remove it after this call. That means the next call also include the sub-directory from the last directory.

That's the solution. After the change everything works as expected.

Thank you and @infratec as well!

camille · Post by **camille** » Mon Dec 14, 2020 10:35 pm

I'm afraid I have to ask a follow up question...

I've added a count mechanism to the ScanDir function so that I'm able to let it use a PostEvent() on start and when it ends and in the main loop two Cases to track these states

Code: Select all

        Case #EventBeginProcessing
          Debug "Thread begin processing"


        Case #EventProcessingFinished
          Debug "Thread processing finished"
          Debug "Map size: " + MapSize(SSDThread\ItemInfos())

Now I'm starting the thread two times but wait until the first has finished.

Code: Select all

  Define.i thread_SimpleScanDir
  Define SSDThread.SimpleScanDir::SSD_PARAMS
  ClearMap(SSDThread\ItemInfos())
  SSDThread\path     = "D:\Install"
  SSDThread\pattern  = "*.*"
  SSDThread\cntCalls = 0

  thread_SimpleScanDir = CreateThread(SimpleScanDir::@ScanDir(), SSDThread)
  If thread_SimpleScanDir
    Repeat
      If IsThread(thread_SimpleScanDir) = 0
        Break
      EndIf
    ForEver
  EndIf
  Debug "no event 1: " + MapSize(SSDThread\ItemInfos())

  ClearMap(SSDThread\ItemInfos())
  SSDThread\path     = "D:\Temp"
  SSDThread\pattern  = "*.*"
  SSDThread\cntCalls = 0

  thread_SimpleScanDir = CreateThread(SimpleScanDir::@ScanDir(), SSDThread)
  Delay(50)
  Debug "no event 2: " + MapSize(SSDThread\ItemInfos())

The second thread will only run for about 5 ms (there are only some files in that "D:\Temp" folder so I'm using a static delay time for testing here.

Ok, running this leads to the correct output for the map size after the threads have ended (no event 1 / 2:)
But NOT in the main event loop (Map size:)!

Code: Select all

no event 1: 2557
no event 2: 47
Thread begin processing
Thread processing finished
Map size: 47
Thread begin processing
Thread processing finished
Map size: 47

Both values there are only from the second thread...

Is this a "the main loop didn't catch the PostEvent() message from the first thread fast enough" problem or what am I missing here?

Thank you,
Camille

infratec · Post by **infratec** » Mon Dec 14, 2020 11:13 pm

Code: Select all

  Define.i thread_SimpleScanDir
  Define SSDThread.SimpleScanDir::SSD_PARAMS
  ClearMap(SSDThread\ItemInfos())
  SSDThread\path     = "D:\Install"
  SSDThread\pattern  = "*.*"
  SSDThread\cntCalls = 0

  thread_SimpleScanDir = CreateThread(SimpleScanDir::@ScanDir(), SSDThread)
  If IsThread(thread_SimpleScanDir)
    WaitThread(thread_SimpleScanDir)
  EndIf
  Debug "no event 1: " + MapSize(SSDThread\ItemInfos())

  ClearMap(SSDThread\ItemInfos())
  SSDThread\path     = "D:\Temp"
  SSDThread\pattern  = "*.*"
  SSDThread\cntCalls = 0

  thread_SimpleScanDir = CreateThread(SimpleScanDir::@ScanDir(), SSDThread)
  If IsThread(thread_SimpleScanDir)
    WaitThread(thread_SimpleScanDir)
  EndIf
  Debug "no event 2: " + MapSize(SSDThread\ItemInfos())

Without a working/not working code I can not tell you more.
Snippets are not enough to see what you have done.

camille · Post by **camille** » Mon Dec 14, 2020 11:48 pm

I've stripped everything for the bare minimum...

Thread_SSD.pb

Code: Select all


XIncludeFile "enumerations.pbi" : UseModule Enums
XIncludeFile "SimpleScanDir_Threaded.pbi"

Define.i thread_SimpleScanDir
SSDThread.SimpleScanDir::SSD_PARAMS
ClearMap(SSDThread\ItemInfos())
SSDThread\path     = "D:\Install"
SSDThread\pattern  = "*.*"
SSDThread\cntCalls = 0

thread_SimpleScanDir = CreateThread(SimpleScanDir::@ScanDir(), SSDThread)
If IsThread(thread_SimpleScanDir)
  WaitThread(thread_SimpleScanDir)
EndIf
Debug "no event 1: " + MapSize(SSDThread\ItemInfos())

ClearMap(SSDThread\ItemInfos())
SSDThread\path     = "D:\Temp"
SSDThread\pattern  = "*.*"
SSDThread\cntCalls = 0

thread_SimpleScanDir = CreateThread(SimpleScanDir::@ScanDir(), SSDThread)
If IsThread(thread_SimpleScanDir)
  WaitThread(thread_SimpleScanDir)
EndIf
Debug "no event 2: " + MapSize(SSDThread\ItemInfos())


OpenWindow(0, 0, 0, 200, 100, "Test", #PB_Window_ScreenCentered|#PB_Window_SystemMenu)

Repeat
  Select WaitWindowEvent()
    Case #PB_Event_CloseWindow
      End

    Case #EventBeginProcessing
      Debug "Thread begin processing"

    Case #EventProcessingFinished
      Debug "Thread processing finished"
      Debug "Map size: " + MapSize(SSDThread\ItemInfos())

  EndSelect
ForEver

enumerations.pbi

Code: Select all


DeclareModule Enums

  Enumeration #PB_Event_FirstCustomValue
    #EventBeginProcessing
    #EventProcessingFinished
  EndEnumeration

EndDeclareModule

Module Enums
EndModule

SimpleScanDir_Threaded.pbi

Code: Select all


DeclareModule SimpleScanDir

  Structure SSD_PARAMS
    Map ItemInfos.i()
    path.s
    pattern.s
    cntCalls.q
  EndStructure

  NewMap PathExclusions.s()
  PathExclusions(".") = ""
  PathExclusions("..") = ""

  CompilerIf #PB_Compiler_OS = #PB_OS_Windows
    PathExclusions("$recycle.bin") = ""
    PathExclusions("system volume information") = ""
  CompilerEndIf

  Declare.i ScanDir(*SSDThread.SSD_PARAMS)

EndDeclareModule

; *************************************************************************************************

Module SimpleScanDir

  EnableExplicit

  Procedure.i ScanDir(*SSDThread.SSD_PARAMS)
    Protected.i hDir
    Protected.s path

    If *SSDThread\cntCalls = 0
      *SSDThread\cntCalls = 1
      PostEvent(Enums::#EventBeginProcessing)
    EndIf

    ; This needs to be kept local!
    path = *SSDThread\path

    If path <> ""
      ; path requires a trailing backslash!
      If Right(path, 1) <> #PS$ : path + #PS$ : EndIf

      hDir = ExamineDirectory(#PB_Any, path, *SSDThread\pattern)
      If hDir
        While NextDirectoryEntry(hDir)
          Select DirectoryEntryType(hDir)
            Case #PB_DirectoryEntry_Directory
              ; Skip on defined entries
              If FindMapElement(SimpleScanDir::PathExclusions(), LCase(DirectoryEntryName(hDir)))
                Continue
              EndIf

              ; Recurse into subfolder(s)
              *SSDThread\cntCalls + 1
              *SSDThread\path = path + DirectoryEntryName(hDir) + #PS$
              ScanDir(*SSDThread.SSD_PARAMS)

            Case #PB_DirectoryEntry_File
              *SSDThread\ItemInfos(path + DirectoryEntryName(hDir)) = DirectoryEntryDate(hDir, #PB_Date_Created)
          EndSelect
        Wend
        FinishDirectory(hDir)
      EndIf

      *SSDThread\cntCalls - 1
      If *SSDThread\cntCalls = 0
        PostEvent(Enums::#EventProcessingFinished)
      EndIf
    EndIf

  EndProcedure

EndModule

infratec · Post by **infratec** » Tue Dec 15, 2020 8:14 am

I think you did not fully understand what a thread is, and how it is working

Code: Select all

EnableExplicit

XIncludeFile "enumerations.pbi"
XIncludeFile "SimpleScanDir_Threaded.pbi"

Define.i thread1, thread2
Define SSDThread1.SimpleScanDir::SSD_PARAMS, SSDThread2.SimpleScanDir::SSD_PARAMS
Define *SSDThreadPtr.SimpleScanDir::SSD_PARAMS

UseModule Enums


ClearMap(SSDThread1\ItemInfos())
SSDThread1\path     = "c:\tmp"
SSDThread1\pattern  = "*.*"
SSDThread1\cntCalls = 0
SSDThread1\thread = CreateThread(SimpleScanDir::@ScanDir(), @SSDThread1)


ClearMap(SSDThread2\ItemInfos())
SSDThread2\path     = "c:\Temp"
SSDThread2\pattern  = "*.*"
SSDThread2\cntCalls = 0
SSDThread2\thread = CreateThread(SimpleScanDir::@ScanDir(), @SSDThread2)


OpenWindow(0, 0, 0, 200, 100, "Test", #PB_Window_ScreenCentered|#PB_Window_SystemMenu)

Repeat
  Select WaitWindowEvent()
    Case #PB_Event_CloseWindow
      End

    Case #EventBeginProcessing
      Debug "Thread begin processing: " + Str(EventData())

    Case #EventProcessingFinished
      Debug "Thread processing finished:" + Str(EventData())
      *SSDThreadPtr = EventData()
      Debug "Map size: " + MapSize(*SSDThreadPtr\ItemInfos())

  EndSelect
ForEver

For that you need:

Code: Select all

Structure SSD_PARAMS
    Thread.i
    Map ItemInfos.i()
    path.s
    pattern.s
    cntCalls.q
 EndStructure

...

PostEvent(Enums::#EventBeginProcessing, 0, 0, 0, *SSDThread)

...

PostEvent(Enums::#EventProcessingFinished, 0, 0, 0, *SSDThread)

And you need, of course, thead safe enabled in compiler options.

camille · Post by **camille** » Thu Dec 17, 2020 10:01 am

Thanks a lot @infratec,

I hope I've got it now

PureBasic Forums - English

Recursive thread function?

Recursive thread function?

Re: Recursive thread function?

Re: Recursive thread function?

Re: Recursive thread function?

Re: Recursive thread function?

Re: Recursive thread function?

Re: Recursive thread function?

Re: Recursive thread function?

Re: Recursive thread function?

Re: Recursive thread function?

Re: Recursive thread function?