Repository navigation
Description
What are you really trying to do?
We run a short-lived Python script in CI. It starts a time-skipping WorkflowEnvironment, shuts it down, prints a result, and exits. A shell captures its output with $(...), so the CI step waits until the process exits.
Describe the bug
Sometimes the process does not exit. asyncio.run() returns and the script prints its last line, but interpreter finalization never completes. The CI step then waits until its time limit (120 min in one case).
gdb on a hung process shows a deadlock between the Runtime drop and a Tokio blocking-pool thread:
- Main thread:
_Py_Finalize, then the final GC, thentp_deallocofRuntimeRef, then the drop ofCoreRuntimeHandle/CoreRuntime, thenBlockingPool::drop. The drop waits for the blocking-pool threads to finish. - Tokio blocking-pool thread: a
BlockingTaskofpyo3_async_runtimes::generic::future_into_py_with_localsis inpyo3_async_runtimes::generic::set_result. That callsloop.call_soon_threadsafe, which writes to the event-loop self-pipe (sock_send), and then reacquires the GIL (PyEval_RestoreThread>take_gil). The interpreter is finalizing, so CPython parks this thread forever inPyThread_hang_thread.
The drop waits for a thread that never returns, so the process never exits.
PR #1998 (in 1.35.0) releases the GIL while the Core runtime is dropped. That does not prevent this hang: during finalization, CPython parks any non-main thread that asks for the GIL, whether the GIL is free or not. 1.35.0 still hangs.
Trimmed stacks from temporalio 1.35.0, Python 3.13.11, Linux aarch64:
Thread 2 "tokio-rt-worker":
#0 __libc_pause ()
#1 PyThread_hang_thread () at Python/thread_pthread.h:440
#2 take_gil (tstate=...) at Python/ceval_gil.c:314
#3 _PyEval_AcquireLock (tstate=...) at Python/ceval_gil.c:593
#4 _PyThreadState_Attach (tstate=...) at Python/pystate.c:2086
#5 PyEval_RestoreThread (tstate=...) at Python/ceval_gil.c:659
#6 sock_call_ex (... sock_func=<sock_send_impl> ...) at ./Modules/socketmodule.c
#8 sock_send (...) at ./Modules/socketmodule.c:4414
...
#17 pyo3_async_runtimes::generic::set_result () from temporal_sdk_bridge.abi3.so
#18 tokio::runtime::task::raw::poll::<tokio::runtime::blocking::task::BlockingTask<pyo3_async_runtimes::generic::future_into_py_with_locals<temporal_sdk_b...
#19 std::sys::backtrace::__rust_begin_short_backtrace::<<tokio::runtime::blocking::pool::Spawner>::spawn_thread::{closure#0}, ()> ()
Thread 1 "python":
#0 syscall ()
#1 <parking_lot::condvar::Condvar>::wait_until_internal ()
#2 <tokio::runtime::park::Inner>::park ()
#3 <tokio::runtime::blocking::pool::BlockingPool as core::ops::drop::Drop>::drop ()
#5 core::ptr::drop_glue::<temporalio_sdk_core::CoreRuntime> ()
#6 core::ptr::drop_glue::<temporal_sdk_bridge::runtime::CoreRuntimeHandle> ()
#7 <alloc::sync::Arc<temporal_sdk_bridge::runtime::CoreRuntimeHandle>>::drop_slow ()
#8 core::ptr::drop_glue::<temporal_sdk_bridge::runtime::Runtime> ()
#11 pyo3::impl_::pyclass::tp_dealloc::<temporal_sdk_bridge::runtime::RuntimeRef> ()
#12 _Py_Dealloc (op=<builtins.RuntimeRef ...>) at Objects/object.c:2939
...
gc_collect_main (... reason=_Py_GC_REASON_SHUTDOWN) at Python/gc.c:1421
_PyGC_CollectNoFail () at Python/gc.c:1657
_Py_Finalize () at Python/pylifecycle.c:2134
Py_RunMain () at Modules/main.c:777
1.13.0 shows the same two stacks. Its pool drop is reached through CoreRuntime directly, without CoreRuntimeHandle.
Minimal Reproduction
repro.py:
import asyncio
import sys
from temporalio.testing import WorkflowEnvironment
async def main() -> None:
env = await WorkflowEnvironment.start_time_skipping(
test_server_existing_path=sys.argv[1]
)
await env.shutdown()
asyncio.run(main())
print("main returned", flush=True)Run it in a loop in a Linux container. This needs a test-server binary that is already downloaded; the first block below downloads one into /tmp/tts.
docker run --rm -v "$PWD/repro.py:/repro.py" ghcr.io/astral-sh/uv:python3.13-bookworm bash -c '
uv venv -q /venv --python 3.13
uv pip install -q --python /venv/bin/python temporalio==1.35.0
mkdir -p /tmp/tts
/venv/bin/python - <<EOF
import asyncio
from temporalio.testing import WorkflowEnvironment
async def main():
env = await WorkflowEnvironment.start_time_skipping(download_dest_dir="/tmp/tts")
await env.shutdown()
asyncio.run(main())
EOF
BIN=$(ls /tmp/tts/* | grep -v downloading | head -1)
for i in $(seq 1 200); do
timeout 60 /venv/bin/python /repro.py "$BIN" > /tmp/out.txt
rc=$?
if [ $rc -ne 0 ]; then echo "run $i: exit $rc (124 = hung), stdout: $(cat /tmp/out.txt)"; break; fi
done'A hung run prints main returned and then times out (exit code 124). To get the stacks, run the container with --cap-add SYS_PTRACE, install gdb, and run gdb -p <pid> -batch -ex "thread apply all bt" on the hung process.
Environment/Versions
- OS and processor: Linux aarch64 (Docker Desktop on an Apple Silicon Mac, image
ghcr.io/astral-sh/uv:python3.13-bookworm). Also seen in CI on Linux x86_64 (Atlassian Bitbucket Pipelines). - Python: 3.13.11 (container), 3.13.15 (CI).
- Temporal version: temporalio 1.13.0 and 1.35.0, with the time-skipping test server that each version downloads.
- Not reproduced on macOS arm64 with temporalio 1.13.0 (0 hangs in 40 runs).
Observed hang frequency, cached test-server binary:
| temporalio | Script | Hangs |
|---|---|---|
| 1.13.0 | Same start/shutdown as repro.py, inside a small CLI |
3 in 28 runs |
| 1.35.0 | Same CLI | 1 in 60 runs |
| 1.35.0 | repro.py above, two loops that each stopped at the first hang |
2 in 29 runs (runs 5 and 24) |
A script that also runs a Worker and a failing workflow before it exits hangs the same way (2 in 11 runs, 1.13.0).
Additional context
Proposed fix (we can open a PR): in CoreRuntimeHandle::drop, check sys.is_finalizing() while attached, and leak the Core runtime with std::mem::forget when it is true. Python::try_attach returns Some on the finalizing thread, so its result does not show this state; a version that leaked only when try_attach returned None still hung 4 times in 76 runs. With the sys.is_finalizing() check, a source build of 1.35.0 had 0 hangs in 200 runs, and tests/test_runtime.py gave the same results as the unchanged build.
Our workaround: the script flushes stdout and stderr and calls os._exit(status) after its result, so finalization never runs. Over 150 runs it exited every time.
Related: #1998 (GIL released during the Core runtime drop), #300 (PyGILState_Release fatal error during finalization), and possibly #834 (a hang after a pytest run on Python 3.13, closed without a root cause).
Activity
AppSolves commentedon Nov 9, 2026
Hi! We've independently encountered what appears to be the same interpreter-finalization deadlock with temporalio 1.35.0 on Linux x86_64 and CPython 3.13.
Our native GDB captures show the main thread in CoreRuntimeHandle/Tokio shutdown while another native thread is blocked acquiring the Python GIL, including PyThread_hang_thread.
We built and tested the proposed fix from #2047 in an isolated custom wheel. The finalization path looks promising, but we discovered a potential issue:
When sys.is_finalizing() raises an exception, the patch retains the CoreRuntime. In our fault-injection test, three consecutive runtime creations and drops left 2, 4 and 6 native threads alive, whereas the official SDK left none.
We have not yet established a causal RED/GREEN reproduction of the original deadlock with the patched wheel.
Could you clarify why #2047 was closed? Is there a preferred or planned upstream solution for safely handling runtime destruction during Python finalization?
We'd be happy to contribute a minimal fix and additional sanitized test evidence.
Thanks!