Threads - whats wrong?

Pantcho!! · Post by **Pantcho!!** » Tue Feb 24, 2004 9:00 pm

Excuse me! when i said i cant open more then 10 windows i didnt mean i didnt see the CONST vars!!!
i meant when i change the TOTAL CONST var to more then 10 like 20
it wont open the window and give me an err in the debugger so relax
i am not that dumb.

now i am releaxed.

i tried to get this code to work with network with no luck
it seems when u use multi threads and multi connections pb cant handle it
that sucks.

if someone here can do this great IRIA example and implent a a network
routine in the DOTASK procedure THAT WORKS that would be AWESOME!
we could start doing a real SERVER and more.
come on guys lets put the thread subject ONTOP!! like i like it!

cya for now

NoahPhense · Post by **NoahPhense** » Tue Feb 24, 2004 9:25 pm

Pantcho!! wrote:Excuse me! when i said i cant open more then 10 windows i didnt mean i didnt see the CONST vars!!!
i meant when i change the TOTAL CONST var to more then 10 like 20
it wont open the window and give me an err in the debugger so relax
i am not that dumb.

now i am releaxed.

i tried to get this code to work with network with no luck
it seems when u use multi threads and multi connections pb cant handle it
that sucks.

if someone here can do this great IRIA example and implent a a network
routine in the DOTASK procedure THAT WORKS that would be AWESOME!
we could start doing a real SERVER and more.
come on guys lets put the thread subject ONTOP!! like i like it!

cya for now

One thing I've learned so far about PB, is that not much is impossible.

I've seen things done in here that were said to never work. A week or
two later, someone posted the solution. It's a matter of the right people
reading 'these threads'..

Well, off to stupid work.. catch yall laters..

- np

Pantcho!! · Post by **Pantcho!!** » Wed Feb 25, 2004 2:14 am

I've searched

didnt found ANY webserver or NETWORK source related to Threads.

8O

Iria · Post by **Iria** » Wed Feb 25, 2004 4:14 pm

As you've probably guesses I like to play about with this stuff.

Here is my vastly simplified and hopefully significantly improved version of my previous example.

Ive taken the time to think this one through and I believe it would be a reasonable place to start if you wish to schedule background tasks.

enjoy...

IRIA

Code: Select all



; Thread Pool with Queues - using PureBasic by IRIA Feb 2004
; ==========================================================
;
; First of all please comment on or propose improvements to this code!
;
; Second of all this code was tested agains PB 3.81
;
; Thirdly, PureBasic is not thread safe. If you need critical thread safetly use
; the underlying OS API calls provided.
;
; NOTE: This code may or may not work, may or may not be good at what it does, I take
; no responsibility for its use or behaviour during use. Use it at your own risk.
;
; Set the global values to sensible settings, it may be cool too see 200 threads try to run
; at the same time but chances are you will break your system.
;
; If you can avoid the use of threads, I recommend that you do.
;
; Try to imagine sensible scenarios, for example, rendering a complex filter on an image
; listening in the background for network connections, or compression/encryption etc.
; most of these dont require a pool of threads working on tasks.
; Sometimes though you want to say spawn 100 connections to an Email server and 
; need a way of metering the threads, this code gives an example of how you might 
; allocate threads to undertake a task in an orderly fashion, and provides some example
; techinques for avoiding threads accessing the same data, and waiting for other threads 
; to complete before starting.
;
; The code cannot make allowance for PB not having thread safe commands.
;
; hope its of some use.



#TOTAL_TASKS = 100            ; total number of tasks to action 
#THREAD_POOL_SIZE = 15        ; Max size of the thread pool so value + 1
#PAUSED = 0               
#PROGRESS_TIME = 100

Structure TaskInfo
  TaskID.l
  ThreadID.l
  ThreadParentID.l
  UniqueGadgetWindowID.l
  RandomTaskTime.l
EndStructure

Dim Task.TaskInfo(#TOTAL_TASKS)          ; tracks usage of each tasks private data

Procedure ExecuteTask(TaskID.l)
  
  ; only start working if our parent in the same queue has finished
  ; this should prevent the threads who are sharing the same thread queue 
  ; overlapping MAKE SURE THIS IS THE FIRST THING YOUR TASK DOES!
  WaitThread(Task.TaskInfo(TaskID)\ThreadParentID) 
  
  ; ENSURE YOUR TASK ONLY STORES AND READS DATA FROM ITS OWN STRUCUTRE SPACE IN THE GLOBAL
  ; TASK ARRAY - this should limit the ability for data read/write contention between
  ; tasks running on threads
  Task.TaskInfo(TaskID)\UniqueGadgetWindowID = TaskID
  Task.TaskInfo(TaskID)\RandomTaskTime = Random(100)+#PROGRESS_TIME
   
  ; gadgets and windows which number = zero are not possible to open
  OpenWindow(Task.TaskInfo(TaskID)\UniqueGadgetWindowID, 200+Random(400), 200+Random(400), 300, 100, #PB_Window_SystemMenu , " Task Number: "+Str(TaskID))
  
  CreateGadgetList(WindowID(Task.TaskInfo(TaskID)\UniqueGadgetWindowID)) 
  ProgressBarGadget(Task.TaskInfo(TaskID)\UniqueGadgetWindowID, 10, 10, 280, 50, 0, Task.TaskInfo(TaskID)\RandomTaskTime, #PB_ProgressBar_Smooth) 
  exit = #False
  
  Repeat 
    eventid = WindowEvent()
    If GetGadgetState(Task.TaskInfo(TaskID)\UniqueGadgetWindowID) < Task.TaskInfo(TaskID)\RandomTaskTime
      SetGadgetState(Task.TaskInfo(TaskID)\UniqueGadgetWindowID, GetGadgetState(Task.TaskInfo(TaskID)\UniqueGadgetWindowID) + 1)  
    Else 
      CloseWindow(Task.TaskInfo(TaskID)\UniqueGadgetWindowID)
      exit = #True
    EndIf
    Delay(1)
  Until exit = #True 
  
  
EndProcedure

Procedure QueueController()
  
  ; thread that controls start of our childrent task threads
  ; waits a second so the pause issued bellow catches it doing nothing
  Delay(1000)
  
  ; Were off!
  Debug "Controller Queue Thread Closed"
  
EndProcedure


Procedure PreScheduleAllTasks()
  
  ; this routine will now divide the number of tasks by the number of queues available
  ; assign a task to each queue, the first task in each queue will have Queue Controller as
  ; its parent.
  ;
  ; Any thread assigned to work on a task with Queue Controller parent will start paused,
  ; all other threads assigned to a task with a parent thread, will wait for the parent thread to
  ; finish before starting, effectively a cascade
  
  ; Queue Master Controller
  QueueMasterControllerThreadID.l = CreateThread(@QueueController(),#NULL)
  
  ; Immediately pause this thread!
  PauseThread(QueueMasterControllerThreadID)
  
  If #THREAD_POOL_SIZE > #TOTAL_TASKS
    ; more queues than tasks, not ideal but possible
    Debug "Sort your pool size and task sizes out!"
  Else 
    QueueCounter = 0
    For TaskIndex = 0 To #TOTAL_TASKS
      
      ; setup the tasks
      Task.TaskInfo(TaskIndex)\TaskID = TaskCounter
      
      
      ; if this is first task in queue set its parent to the master queue controller
      If QueueCounter <= #THREAD_POOL_SIZE
        Task.TaskInfo(TaskIndex)\ThreadParentID = QueueMasterControllerThreadID
      Else
        ; the parent of a task is the thread pool size offset back into the array
        Task.TaskInfo(TaskIndex)\ThreadParentID = Task.TaskInfo(TaskIndex - #THREAD_POOL_SIZE)\ThreadID
      EndIf
      
      ; kick off the next thread task
      Task.TaskInfo(TaskIndex)\ThreadID =  CreateThread(@ExecuteTask(),TaskIndex)
      
      ; move on to next queue
      QueueCounter + 1
      
    Next TaskIndex
  EndIf
  
  ; Ok now we kick off everything by resuming the control thread
  ResumeThread(QueueMasterControllerThreadID)
  
EndProcedure



;Main 

Debug "Program Start"

PreScheduleAllTasks()

Debug "Program End"

Debug "Infinite Loop"

Repeat
  
  ; this would be where your main code runs, it may for example be a GUI which
  ; looks at the global array for update info and displays this to the user.
  Delay (1)

ForEver

Post by **Dare2** » Wed Feb 25, 2004 4:32 pm

Enjoying ...

Thanks again.

NoahPhense · Post by **NoahPhense** » Wed Feb 25, 2004 6:10 pm

Iria wrote:As you've probably guesses I like to play about with this stuff.

Here is my vastly simplified and hopefully significantly improved version of my previous example.

Ive taken the time to think this one through and I believe it would be a reasonable place to start if you wish to schedule background tasks.

enjoy...

IRIA

Very kewl. This will come in hand for my batch processing.

But we should keep in mind. That multithreading will truely
never exist unless we have more than one processor.

Which brings to mind my friends server.. he's got (2) quad xeon's.

- np

Pantcho!! · Post by **Pantcho!!** » Wed Feb 25, 2004 6:32 pm

IRIA!
thanks man.

is there any chance if u totaly go for it... try to use network in the main threaded procedure... cuz i tried with no luck.
it very important because that was my main goal.

Iria · Post by **Iria** » Wed Feb 25, 2004 10:36 pm

PB network commands appear to be very flakey when tyring to run from within a thread...still have a few ideas to try but dont hold out hope on them!.

Meanwhile I did find that some OS API calls are thread safe and will happily work for network connections. Ive modified my example to use some realmedia productions FTP API calls so you can see.

Just make sure you have a c:\

and run this:

Code: Select all

; Thread Pool with Queues - using PureBasic by IRIA Feb 2004
; ==========================================================
;
; First of all please comment on or propose improvements to this code!
;
; Second of all this code was tested agains PB 3.81
;
; Thirdly, PureBasic is not thread safe. If you need critical thread safetly use
; the underlying OS API calls provided.
;
; NOTE: This code may or may not work, may or may not be good at what it does, I take
; no responsibility for its use or behaviour during use. Use it at your own risk.
;
; Set the global values to sensible settings, it may be cool too see 200 threads try to run
; at the same time but chances are you will break your system.
;
; If you can avoid the use of threads, I recommend that you do.
;
; Try to imagine sensible scenarios, for example, rendering a complex filter on an image
; listening in the background for network connections, or compression/encryption etc.
; most of these dont require a pool of threads working on tasks.
; Sometimes though you want to say spawn 100 connections to an Email server and 
; need a way of metering the threads, this code gives an example of how you might 
; allocate threads to undertake a task in an orderly fashion, and provides some example
; techinques for avoiding threads accessing the same data, and waiting for other threads 
; to complete before starting.
;
; The code cannot make allowance for PB not having thread safe commands.
;
; hope its of some use.



#TOTAL_TASKS = 10           ; total number of tasks to action 
#THREAD_POOL_SIZE = 2        ; Max size of the thread pool so value + 1
#PAUSED = 0               
#PROGRESS_TIME = 100

Structure TaskInfo
  TaskID.l
  ThreadID.l
  ThreadParentID.l
  UniqueGadgetWindowID.l
  RandomTaskTime.l
EndStructure

Dim Task.TaskInfo(#TOTAL_TASKS)          ; tracks usage of each tasks private data

Procedure ExecuteTask(TaskID.l)
  
  ; only start working if our parent in the same queue has finished
  ; this should prevent the threads who are sharing the same thread queue 
  ; overlapping MAKE SURE THIS IS THE FIRST THING YOUR TASK DOES!
  WaitThread(Task.TaskInfo(TaskID)\ThreadParentID) 
  
  ; some net code borrowed from reelmediaproductions to test net connections work  
  InternetOpen_("FTP",1,"","",0) 
  InternetConnect_(hInternet,Server,Port,User,Password,1,0,0) 
  
  hInternet=InternetOpen_("FTP",1,"","",0) 
  If hInternet 
    
    hConnect=InternetConnect_(hInternet,"ftp.reelmediaproductions.com",21,"guest","guest",1,0,0) 
    If hConnect 
      
      FtpSetCurrentDirectory_(hConnect,"test/")
      hFind=FtpFindFirstFile_(hConnect,"*.*",@FTPFile.WIN32_FIND_DATA,0,0) 
      If hFind 
        Find=1 
        Debug PeekS(@FTPFile\cFileName) 
        While Find 
          Find=InternetFindNextFile_(hFind,@FTPFile) 
          If Find 
            Debug PeekS(@FTPFile\cFileName) 
          EndIf      
        Wend 
      EndIf 
      FTPGetFile_(hConnect,"hello.txt","C:\task-"+Str(TaskID)+"-download.txt",0,0,0,0) 
    EndIf
    InternetCloseHandle_(hInternet) 
  EndIf
  
EndProcedure
    
Procedure QueueController()
  
  ; thread that controls start of our childrent task threads
  ; waits a second so the pause issued bellow catches it doing nothing
  Delay(1000)
  
  ; Were off!
  Debug "Controller Queue Thread Closed"
  
EndProcedure
    
    
Procedure PreScheduleAllTasks()
  
  ; this routine will now divide the number of tasks by the number of queues available
  ; assign a task to each queue, the first task in each queue will have Queue Controller as
  ; its parent.
  ;
  ; Any thread assigned to work on a task with Queue Controller parent will start paused,
  ; all other threads assigned to a task with a parent thread, will wait for the parent thread to
  ; finish before starting, effectively a cascade
  
  ; Queue Master Controller
  QueueMasterControllerThreadID.l = CreateThread(@QueueController(),#NULL)
  
  ; Immediately pause this thread!
  PauseThread(QueueMasterControllerThreadID)
  
  If #THREAD_POOL_SIZE > #TOTAL_TASKS
    ; more queues than tasks, not ideal but possible
    Debug "Sort your pool size and task sizes out!"
  Else 
    QueueCounter = 0
    For TaskIndex = 0 To #TOTAL_TASKS
      
      ; setup the tasks
      Task.TaskInfo(TaskIndex)\TaskID = TaskCounter
      
      
      ; if this is first task in queue set its parent to the master queue controller
      If QueueCounter <= #THREAD_POOL_SIZE
        Task.TaskInfo(TaskIndex)\ThreadParentID = QueueMasterControllerThreadID
      Else
        ; the parent of a task is the thread pool size offset back into the array
        Task.TaskInfo(TaskIndex)\ThreadParentID = Task.TaskInfo(TaskIndex - #THREAD_POOL_SIZE)\ThreadID
      EndIf
      
      ; kick off the next thread task
      Task.TaskInfo(TaskIndex)\ThreadID =  CreateThread(@ExecuteTask(),TaskIndex)
      
      ; move on to next queue
      QueueCounter + 1
      
    Next TaskIndex
  EndIf
  
  ; Ok now we kick off everything by resuming the control thread
  ResumeThread(QueueMasterControllerThreadID)
  
EndProcedure
    
    
    
;Main 
    
Debug "Program Start"
    
Debug "Start the network"

Debug "Start the threads"
PreScheduleAllTasks()

Pantcho!! · Post by **Pantcho!!** » Wed Feb 25, 2004 10:54 pm

SuperB! You Rock man.

thanks allot, that was amazing.
now i can run multiple client in seconds.
thanks allot man.

ShDancer · Post by **ShDancer** » Fri Feb 27, 2004 6:12 pm

Maybe this is out off this topic but after i read it i just couln't stop thinking creating somekind of a manager for multi threads, only in Purebasic without using nothing from the Windows API.

So after 3 hours i've came out with this.

Please can some one help me correct or add some features for this?

ATENTION: This is very Beta, but i would like to get some kind of comments, about it.

Thanks in advance.

Code: Select all

; Multi Thread Lib V 0.00001 very, very, very, very, ...... -> Beta

;
; The idea is to implement a thread that manages a list of threads
; and you can start/stop/pause several that are managed by this one
;

; Not completly implemented or verified
; Need to optimize it very much and correct those many nasty bugs


Enumeration ; Thread Lib Status
  #Started
  #Stopped
EndEnumeration

Enumeration ; Thread Status
  #ThreadStatus_Start
  #ThreadStatus_Stop
  #ThreadStatus_Running
  #ThreadStatus_Paused
  #ThreadStatus_Resume
EndEnumeration

Enumeration ; Thread Priorities
  ; The priority value can go from 1 to 32.
  ;  1 is the lowest priority available
  ; 16 is the normal priority 
  ; 32 is the time critical priority
  #ThreadPriority_Lowest  = 1
  #ThreadPriority_Normal  = 16
  #ThreadPriority_Highest = 32
EndEnumeration

Structure Thread_Poll_Structure
  ThreadID.l
  ThreadDescription.s
  ThreadStatus.b
  ThreadParameter.s
  ThreadPriority.b
  ThreadWaitUntilEnd.b ; Future Implementation/Updates
EndStructure


; Global Constants

#Empty = ""
#ThreadStartError01 = "You must start Multi Thread Lib first."
#ThreadStartError02 = "Multi Thread Lib Couldn't be started."
#ThreadStartError03 = "Garbage collector periodicity value too LOW."

; For Debug Porposes
#Debug = #False

; Global Variables

Global ThreadLibStatus
; Global Thread_Garbadge_Collector
Global Thread_Garbadge_Collector_ID

NewList Thread_Pool.Thread_Poll_Structure()

ThreadLibStatus = #Stopped
Thread_Garbadge_Collector_ID = 0

; Multi Thread Lib - Procedures

Procedure ThreadDebug( Msg.s )
  ;- Not completly implemented or verified
  If #Debug 
    Debug "Error: " + Msg.s
  Else
    MessageRequester("Error", Msg.s, #PB_MessageRequester_Ok)
  EndIf
EndProcedure

Procedure Thread_Create(ThreadDescription.s, ProcedureAddress)
  ;- Not completly implemented or verified
  If (ThreadLibStatus = #Started )
    ResetList(Thread_Pool())
    AddElement(Thread_Pool())
    Thread_Pool()\ThreadID           = CreateThread( ProcedureAddress, #NULL )
    Thread_Pool()\ThreadDescription  = ThreadDescription.s
    Thread_Pool()\ThreadStatus       = #ThreadStatus_Paused
    Thread_Pool()\ThreadParameter    = #Empty
    Thread_Pool()\ThreadPriority     = #ThreadPriority_Normal
    ; Thread_Pool()\ThreadWaitUntilEnd = #True ; Future Updates
  Else
    ThreadDebug(#ThreadStartError01)
  EndIf
EndProcedure

Procedure Set_Thread_Status( ThreadID.l, Status.b )
  ;- Not completly implemented or verified
  If (ThreadLibStatus = #Started )
    ResetList(Thread_Pool())
    While (NextElement(Thread_Pool()))
      If ( Thread_Pool()\ThreadID = ThreadID.l )
        Thread_Pool()\ThreadStatus = Status.b
        If ( Thread_Pool()\ThreadStatus = #ThreadStatus_Paused )
          PauseThread( ThreadID.l )
        EndIf
        LastElement(Thread_Pool())
      EndIf
    Wend
  Else
    ThreadDebug(#ThreadStartError01)
  EndIf
EndProcedure

Procedure.b Get_Thread_Status( ThreadID.l )
  ;- Not completly implemented or verified
  Protected ReturnValue.b
  If (ThreadLibStatus = #Started )
    ReturnValue = 0
    ResetList(Thread_Pool())
    While (NextElement(Thread_Pool()))
      If ( Thread_Pool()\ThreadID = ThreadID.l )
        ReturnValue = Thread_Pool()\ThreadStatus
        LastElement(Thread_Pool())
      EndIf
    Wend
    ProcedureReturn ReturnValue.b
  Else
    ThreadDebug(#ThreadStartError01)
  EndIf
EndProcedure

Procedure.s Get_Thread_Parameter( ThreadID.l )
  ;- Not completly implemented or verified
  Protected ReturnValue.s
  If (ThreadLibStatus = #Started )
    ReturnValue = ""
    ResetList(Thread_Pool())
    While (NextElement(Thread_Pool()))
      If ( Thread_Pool()\ThreadID = ThreadID.l )
        ReturnValue = Thread_Pool()\ThreadParameter
        LastElement(Thread_Pool())
      EndIf
    Wend
  Else
    ThreadDebug(#ThreadStartError01)
  EndIf
  ProcedureReturn ReturnValue
EndProcedure

Procedure Set_Thread_Parameter( ThreadID.l, Parameter.s )
  ;- Not completly implemented or verified
  If (ThreadLibStatus = #Started )
    If ( Get_Thread_Parameter(ThreadID.l) <> Parameter.s )
      ResetList(Thread_Pool())
      While (NextElement(Thread_Pool()))
        If ( Thread_Pool()\ThreadID = ThreadID.l )
          Thread_Pool()\ThreadParameter = Parameter.s
          LastElement(Thread_Pool())
        EndIf
      Wend
    EndIf
  Else
    ThreadDebug(#ThreadStartError01)
  EndIf
EndProcedure

Procedure Get_Thread_Priority( ThreadID.l )
  ;- Not completly implemented or verified
  Protected ReturnValue.b
  If (ThreadLibStatus = #Started )
    ReturnValue = 0
    ResetList(Thread_Pool())
    While (NextElement(Thread_Pool()))
      If ( Thread_Pool()\ThreadID = ThreadID.l )
        ReturnValue = Thread_Pool()\ThreadPriority
        LastElement(Thread_Pool())
      EndIf
    Wend
  Else
    ThreadDebug(#ThreadStartError01)
  EndIf
EndProcedure

Procedure Set_Thread_Priority( ThreadID.l, Priority.b )
  ;- Not completly implemented or verified
  Protected ReturnValue.b
  If (ThreadLibStatus = #Started )
    ReturnValue = 0
    If ( Get_Thread_Priority(ThreadID.l) <> Priority.b )
      ResetList(Thread_Pool())
      While (NextElement(Thread_Pool()))
        If ( Thread_Pool()\ThreadID = ThreadID.l )
          Thread_Pool()\ThreadPriority = Priority.b
          LastElement(Thread_Pool())
        EndIf
      Wend
    EndIf
  Else
    ThreadDebug(#ThreadStartError01)
  EndIf
EndProcedure

Procedure Thread_Garbadge_Collector( Periodicity.w )
  ;- Not completly implemented or verified
  Shared ThreadLibStatus
  Repeat
    Delay(Periodicity.w)

    ResetList(Thread_Pool())
    While (NextElement(Thread_Pool()) And ThreadLibStatus = #Started)

      Select Thread_Pool()\ThreadStatus
        
        Case #ThreadStatus_Stop
          KillThread( Thread_Pool()\ThreadID )
          DeleteElement( Thread_Pool() )
          LastElement( Thread_Pool() )
              
        Case #ThreadStatus_Running
          ThreadPriority( Thread_Pool()\ThreadID, Thread_Pool()\ThreadPriority )
          LastElement(Thread_Pool())
          
        Case #ThreadStatus_Resume
          Thread_Pool()\ThreadStatus = #ThreadStatus_Running
          ResumeThread( ThreadID.l )
          LastElement(Thread_Pool())
          
      EndSelect
    Wend
  Until ThreadLibStatus = #Stopped
EndProcedure

Procedure.b Multi_Thread_Lib_Start(Peridiciodity.w)
  ;- Not completly implemented or verified
  Protected ReturnValue.b
  ReturnValue = 0
  Shared Thread_Garbadge_Collector_ID
  If ( Peridiciodity > 500 ) ; In MiliSeconds
    Thread_Garbadge_Collector_ID = CreateThread(@Thread_Garbadge_Collector(), Peridiciodity.w )
    If ( Thread_Garbadge_Collector = 0 )
      ThreadLibStatus = #Stopped
      ThreadDebug(#ThreadStartError02)
      ReturnValue = 1
    Else
      ThreadLibStatus = #Started
      ReturnValue = 0
    EndIf
  Else
    ThreadDebug(#ThreadStartError03)
  EndIf
  ProcedureReturn ReturnValue
EndProcedure

Procedure Multi_Thread_Lib_Stop()
  ;- Not completly implemented or verified
  Shared ThreadLibStatus
  Shared Thread_Garbadge_Collector_ID

  If ( ThreadLibStatus = #Started )
    ThreadLibStatus = #Stopped
  EndIf
  If ( ThreadLibStatus = #Stopped )
    WaitThread (Thread_Garbadge_Collector_ID )
    KillThread (Thread_Garbadge_Collector_ID )
  EndIf
EndProcedure

Iria · Post by **Iria** » Fri Feb 27, 2004 8:53 pm

Hi,

Looks interesting, you need to provide some simple example usage code for people to understand the basic usage of your proposed library.

You will get more feedback if you do

IRIA

PureBasic Forums - English

Threads - whats wrong?

..

Code improvements

Re: Code improvements

IRIA!

Hmm tried to use the PB network commands...

Looks interesting