multiprocessing.pool.Pool.apply block infinitely when stressed while using maxtasksperchild

[harsh8398]

BPO	47159
Nosy	@pitrou, @applio, @harsh8398
Files	example.py: Example python script which produces the issue

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-03-30.09:53:14.324>
labels = ['3.7', 'type-bug', 'library']
title = 'multiprocessing.pool.Pool.apply block infinitely when stressed while using maxtasksperchild'
updated_at = <Date 2022-03-30.09:53:14.324>
user = 'https://github.com/harsh8398'

bugs.python.org fields:

activity = <Date 2022-03-30.09:53:14.324>
actor = 'harsh8398'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-03-30.09:53:14.324>
creator = 'harsh8398'
dependencies = []
files = ['50706']
hgrepos = []
issue_num = 47159
keywords = []
message_count = 1.0
messages = ['416331']
nosy_count = 3.0
nosy_names = ['pitrou', 'davin', 'harsh8398']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47159'
versions = ['Python 3.7']

[harsh8398]

We have a long-running process, which uses thread pool to get tasks and these threads then schedule the tasks and send them to multiprocessing.Pool (with maxtasksperchild limit set) using apply function. The original codebase is large but I have extracted this logic into a simple script for convenience so that you can see the issue. The issue is that after sometime the apply function call gets blocked infinitely. This only happens when we use maxtasksperchild and cause the library for multiple process recreation.

How script works:
Script creates an object of the Manager class which is responsible for creating a pool of threads and creating multiprocessing pool. Each thread is provided jobs to execute which is done by Scheduler.get_ready_jobs function call. I have designed this function in a way that 100 jobs are returned after every 5 calls to this function else it returns no jobs. Each thread while performing job sends processing_func to multiprocessing pool for execution which just prints "processing something" and exits.

Environment:
Python 3.7.10
Clang 12.0.5
MacOS v11.6 intel

Steps to reproduce:

1. Download the attached Python script and execute it
2. Let it run for 10-15 seconds and interrupt the execution

You will notice that some threads will exit successfully but for some, you will see the last message printed is "executing cpu_pool apply <thread_name>" which indicates these threads are blocked even though the actual processing_func is just calling a print function.

Warning: You will also not be able to exit the process without killing it.

As a note, this is my first Python issue report so please let me know if you need more information.

[harsh8398]

If this finding helps, this issue does not get reproduced when Python v3.8 is used.