Script using ProactorEventLoop does not exit cleanly when subprocesses are spawned.

[cr1901]

BPO	37381
Nosy	@pfmoore, @tjguk, @asvetlov, @zware, @1st1, @zooba, @cr1901

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-06-23.22:35:46.548>
labels = ['type-bug', '3.7', 'OS-windows', 'expert-asyncio']
title = 'Script using ProactorEventLoop does not exit cleanly when subprocesses are spawned.'
updated_at = <Date 2019-06-24.01:07:24.313>
user = 'https://github.com/cr1901'

bugs.python.org fields:

activity = <Date 2019-06-24.01:07:24.313>
actor = 'cr1901'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows', 'asyncio']
creation = <Date 2019-06-23.22:35:46.548>
creator = 'cr1901'
dependencies = []
files = []
hgrepos = []
issue_num = 37381
keywords = []
message_count = 2.0
messages = ['346345', '346346']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'asvetlov', 'zach.ware', 'yselivanov', 'steve.dower', 'cr1901']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37381'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[cr1901]

On Windows 7, when using ProactorEventLoop, when asyncio.subprocess.terminate is called on the last subprocess when multiple subprocesses are spawned*, the last subprocess will never exit, and the Python interpreter will never exit.

Spawning an instance of "TASKKILL /F" to kill the subprocess, and then awaiting asyncio.subprocess.wait is a workaround. Since both "TASKKILL /F" and asyncio.subprocess.terminate ultimately call TerminateProcess, I would expect the same behavior using both (without the need to wait for the subprocess to actually die).

I have confirmed this behavior on Python 3.5, 3.6, and 3.7 on both MSVC and MinGW Python (although I most frequently use the latter).

Zip file is too large, so I have a GH repo (ready to run) demonstrating this issue: https://github.com/cr1901/proactor

An example run looks like the following:$ python3 proactor.py
Infinite Loop, Iteration 0
Finite Loop, Iteration 0
Infinite Loop, Iteration 1
Finite Loop, Iteration 1
Infinite Loop, Iteration 2
Finite Loop, Iteration 2
Finite Loop, Iteration 3
Infinite Loop, Iteration 3
Infinite Loop, Iteration 4
Task poller finished.

The interpreter hangs after the "Task poller finished." line, which makes me believe that the Python interpreter is trying and failing to cleanup resources for some reason (IOCP never completes?).

*Each subprocess is associated with its own coroutine to handle standard output.

[cr1901]

Update: It turns out multiple spawned subprocesses are NOT required. If I replace "(short_fut, _) = await spawn()" with "short_fut = asyncio.ensure_future(asyncio.sleep(4))", the hang still occurs.

[arhadthedev]

If this issue is fixed by gh-92841, it should be closed.

[kumaraditya303]

@cr1901 Can you check if this is still an issue on main branch and 3.11? It would be better if you can provide a reproducer without involving rust.

[cr1901]

@kumaraditya303 I cannot duplicate this on 3.10.7; I'm not in a position anymore to compile and test the main branch of Python (or 3.11) at present.

There is still a minor bug where if I run the script before both echo processes have terminated, the script runs a single iteration and exits with code 120. I can't tell if it's related to this bug or a separate one.

It would be better if you can provide a reproducer without involving rust.

I don't remember the rationale here. I think I was having trouble spawning a python process taking a script as input. I'll see what I can do to make the reproducer Python-only.

[kumaraditya303]

asyncio subprocess is significantly improved in 3.11 and 3.12 (main). A lot of bugs have been fixed so it is likely the bug is already fixed. I look forward to a Python only reproducer.

[kumaraditya303]

I tried your rust reproducer and the bug is fixed in main, 3.11 and 3.10 it doesn't hangs so closing as it is fixed.