Serialize process spawning across threads with a lock#728
Serialize process spawning across threads with a lock#728veeceey wants to merge 2 commits intoMagicStack:masterfrom
Conversation
Replace the RuntimeError-raising check for concurrent process spawning with a threading.Lock that serializes spawns across different event loops running in separate threads. The old approach would fail with "Racing with another loop to spawn a process" when multiple threads tried to spawn processes concurrently, since the global pthread_atfork handlers can only be active for one loop at a time. Now instead of failing, concurrent spawns wait for the lock, allowing them to proceed sequentially. The lock is properly released in all error paths via a finally block. Fixes MagicStack#508
|
Couldn't build locally (shallow clone, no libuv submodule), but the change is straightforward — replacing the The key points:
Looking forward to CI results to confirm this works end-to-end. |
setuptools>=78 removed the bundled pkg_resources module, causing ModuleNotFoundError on all CI jobs except Python 3.8-ubuntu (which uses an older setuptools). Use packaging.requirements.Requirement instead, which provides the same version-specifier matching and is always available as a dependency of setuptools.
|
The CI failures were not caused by the process-spawning changes in this PR. All 14 failing jobs (everything except Python 3.8-ubuntu) hit the same error during This is because The 3.8-macos failure is a separate, pre-existing flaky test in I pushed a fix that replaces |
When multiple threads each run their own event loop and try to spawn processes concurrently, they race on the global
pthread_atforkhandlers, which can only be active for one loop at a time. The current code detects this race and raisesRuntimeError("Racing with another loop to spawn a process"), which isn't great — it means users have to build their own retry logic or serialization.This replaces the error-raising approach with a
threading.Lockthat serializes process spawning across threads. Instead of failing, concurrent spawns just wait their turn. The lock is held only during the critical fork section (setting uppthread_atforkhandlers, callinguv_spawn, and cleaning up), so it doesn't block any longer than necessary.The lock is properly released in all error paths through a
finallyblock that also cleans up the__forkingstate if something goes wrong mid-fork.Repro from the issue now works without errors:
Fixes #508