How to avoid race conditions and state/stack pollution when creating multiple lua_State in PureBasic with Lua 5.3?

[skinkairewalker]

I’m experimenting with Lua 5.3 embedded in PureBasic By Dadido3 (https://github.com/Dadido3/Lua-PureBasic), and I’ve reached a point where I need to create multiple Lua states (lua_State) in separate threads, so scripts can run in parallel without interfering with each other.

My goal is to be able to create effectively infinite Lua instances (or at least thousands) without running into race conditions, state pollution, or stack corruption, even if they all access some shared variables on the PureBasic side.

Here’s my current example:

Code: Select all

XIncludeFile "Libraries\Lua\Lua.pbi"

UseModule Lua

Global THR_A,THR_B,THR_C,THR_D,THR_E,THR_F,THR_G
Global mutex, counter = 0
#MAX = 100000

mutex = CreateMutex()

ProcedureC PB_Debug(*Lua_State)
  counter + 1
  Debug "LUA STATE [" + Str(*Lua_State) + "] VALUE = " + Str(counter)
  ProcedureReturn 0
EndProcedure

Procedure f_thr_a(*p)
  *Lua_State = luaL_newstate()
  lua_register(*Lua_State, "PB_Sum", @PB_Debug())
  Repeat
    LockMutex(mutex)
      luaL_dostring(*Lua_State, "PB_Sum()")
    UnlockMutex(mutex)
    If counter >= #MAX : ProcedureReturn : EndIf
  ForEver
EndProcedure

; ... repeated for f_thr_b, f_thr_c, etc.

; create threads
THR_A = CreateThread(@f_thr_a(), 0)
THR_B = CreateThread(@f_thr_b(), 0)
THR_C = CreateThread(@f_thr_c(), 0)
THR_D = CreateThread(@f_thr_d(), 0)
THR_E = CreateThread(@f_thr_e(), 0)
THR_F = CreateThread(@f_thr_f(), 0)
THR_G = CreateThread(@f_thr_g(), 0)

; wait for all threads
WaitThread(THR_A)
WaitThread(THR_B)
WaitThread(THR_C)
WaitThread(THR_D)
WaitThread(THR_E)
WaitThread(THR_F)
WaitThread(THR_G)

Debug "END OF PROGRAM"

What I already understand:

• Each lua_State is isolated, so Lua itself isn’t the issue.
• The risk lies in the shared variables on the C/PureBasic side (counter in this example).
• Even with a mutex, locking may become very expensive if I want hundreds or thousands of threads.

What I’d like to know:

• Is there a thread-safe and lightweight way to share simple data (like integers or structs) between multiple lua_State without relying on LockMutex constantly?
• Is it possible to create a Lua state pool in PureBasic (like in C++) to reuse lua_State objects safely?
• Has anyone successfully created many Lua VMs (thousands) in PureBasic without crashes or memory corruption?
• Any techniques to avoid stack pollution when calling luaL_dostring() at high frequency in multiple threads?

End goal:

• Run multiple Lua scripts in parallel, each with its own environment, while still being able to read/write a few global variables on the PureBasic side without locking issues or corruption.
• Any advanced tips on synchronization, using lua_newthread(), or strategies for caching/reusing states would be greatly appreciated.

Thanks in advance!

[tored]

You can use atomic operations for writing to variables from multiple threads without using a mutex.

https://learn.microsoft.com/en-us/windo ... ble-access

[tored]

I'm far from a Lua expert but my understanding of the "Lua way" of doing things is to use coroutines (lua threads). Each coroutine will have their own stack but share global data.

Thus you can have much fewer PB-threads, e.g. 1000 lua threads and 10 PB threads = 10 000 lua threads

With fewer PB threads there will be a much lower pressure on mutexes.

untested prototype code

setup

Code: Select all

Global *L = luaL_newstate()
luaL_openlibs(*L)

luaL_loadstring(*L, "function run() for i=1,3 do print(i) coroutine.yield() end end")
lua_pcall(*L, 0, 0, 0)

NewList coroutinePool.luaptr()

For i = 0 To 999
  *co = lua_newthread(*L)
  lua_getglobal(*co, "run")
  AddElement(coroutinePool())
  coroutinePool() = *co
Next

execute

Code: Select all

ForEach coroutinePool()
  lua_resume(coroutinePool(), 0)
Next

For each PB-thread run the setup, and then execute all coroutines.