multiprocessing Manager error: send() called for a closed connection

@pfmoore

BPO	27749
Nosy	@pfmoore, @tjguk, @bitdancer, @zware, @eryksun, @zooba, @applio, @AraHaan
Files	python.exe.8228.dmp debug_AA1F93C2.JPG: multiprocessing debug Screenshot

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-08-12.18:59:30.581>
labels = ['type-bug', '3.8', '3.9', '3.10', 'library', 'OS-windows']
title = 'multiprocessing Manager error:  send() called for a closed connection'
updated_at = <Date 2021-03-13.22:29:18.086>
user = 'https://bugs.python.org/wevsty'

bugs.python.org fields:

activity = <Date 2021-03-13.22:29:18.086>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2016-08-12.18:59:30.581>
creator = 'wevsty'
dependencies = []
files = ['44099', '44116']
hgrepos = []
issue_num = 27749
keywords = []
message_count = 14.0
messages = ['272557', '272577', '272581', '272584', '272589', '272615', '272616', '272647', '272653', '272743', '272861', '273007', '334464', '334628']
nosy_count = 10.0
nosy_names = ['paul.moore', 'tim.golden', 'r.david.murray', 'zach.ware', 'eryksun', 'steve.dower', 'davin', 'Decorater', 'wevsty', 'Michael Jacob2']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27749'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

When I use Python 3.5.2 some of the time will be a crash.displayed crash module is python35.dll.
I run Python environment is Windows Server 2012R2.I think crash dump maybe will be helpful.Report.wer and carsh dump can be downloaded via the following link.
http://o8z0k9748.bkt.clouddn.com/Python_crash_Dump_20160813_AF1A29AF.7z

I have another question when i using managers lib multiprocessing Maybe have exception.
my code is：
Session_lock.G_Process_Lock = multiprocessing.Manager().Lock()
def execute_select_command(command):
try:
with Session_lock.G_Process_Lock:
values = []
conn = sqlite3.connect(SQLite_DB_NAME)
cursor = conn.cursor()
try:
cursor.execute(command)
values = cursor.fetchall()
except Exception as e:
pass
finally:
cursor.close()
conn.close()
return values
except Exception as e:
print(e)
traceback.print_exc()
finally:
pass

This code is possible to print out the information.

Traceback (most recent call last):
  File "C:/Users/Administrator/Desktop/Monitor_URL\Monitor_SQL.py", line 44, in execute_select_command
    return values
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 949, in __exit__
    return self._callmethod('release')
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 716, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 280, in _send_bytes
    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
TypeError: WriteFile() argument 1 must be int, not None

This maybe seem like a lib problem.

To diagnose the access violation, it may help if you install the debug binaries and try to reproduce the crash using python_d.exe. Attach the dump file to this issue.

Regarding the multiprocessing question, if its a separate issue you need to file it on its own and nosy davin. At first glance, I don't see why the handle is None in the lock's __exit__ method. After you acquire the lock, try logging the connection information. For example:

    import multiprocessing as mp
    import multiprocessing.util

    def test(lock):
        with lock:
            mp.util.info('test:lock:token: %r' % (lock._token,))
            mp.util.info('test:lock:handle: %r' % (lock._tls.connection._handle,))

    if __name__ == '__main__':
        mp.util.log_to_stderr(mp.util.INFO)
        lock = mp.Manager().Lock()
        p = mp.Process(target=test, args=(lock,))
        p.start()
        p.join()

About the crash,python_d.exe crash dump download address is

http://o8z0k9748.bkt.clouddn.com/python_d_from_vs_20160813_930A2640.7z

Crash dump display,at the

python35_d.dll!_Py_ForgetReference(_object * op) Line 1758 C
python35_d.dll!_Py_Dealloc(_object * op) Line 1786 C
python35_d.dll!tupledealloc(PyTupleObject * op) Line 236 C

I hope this can be helpful to you.

About the multiprocessing question
I modified my code becomes
def execute_select_command(command):
values = []
try:
with Session_lock.G_Process_Lock:
mp.util.info('test:lock:token: %r' % (Session_lock.G_Process_Lock._token,))
mp.util.info('test:lock:handle: %r' % (Session_lock.G_Process_Lock._tls.connection._handle,))
conn = sqlite3.connect(SQLite_DB_NAME,check_same_thread = False)
cursor = conn.cursor()
try:
cursor.execute(command)
values = cursor.fetchall()
except Exception as e:
pass
finally:
cursor.close()
conn.close()
return values
except Exception as e:
print(e)
traceback.print_exc()
mp.util.info('Exception:lock:token: %r' % (Session_lock.G_Process_Lock._token,))
mp.util.info('Exception:lock:handle: %r' % (Session_lock.G_Process_Lock._tls.connection._handle,))
pass
return values
finally:
pass

After run，Console output

[INFO/SpawnPoolWorker-18] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-18] test:lock:handle: 720
[INFO/SpawnPoolWorker-17] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-17] test:lock:handle: 908
[INFO/SpawnPoolWorker-16] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-16] test:lock:handle: 744
WriteFile() argument 1 must be int, not None
[INFO/SpawnPoolWorker-16] Exception:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-2876-0-n6aj8inv', id='d0b0efa7d0')
[INFO/SpawnPoolWorker-16] Exception:lock:handle: 744
Could not find thread pid_2720_id_389212239784

this problem and crash both the use of multiprocessing lib after appears.Maybe these two issues are related.
If you need my full code or need to open a new issue, please let me know.Thinks

Crashes also happen from Recursion if not careful. Idk why but it happens. Although some of it is probably by not using try / catch probably anywhere in the python source code as a safeguard. I always like try / catch even in C++ as you never know when crap might happen to make things crash. A Perfect Example is a Access violation.

(Yes I have this is a Standard for me and it gets me to great quality code)
(this makes me wonder if Windows does not use try / catch anywhere. It would be nice if Microsoft Required try / catch in every function for drivers as well to hopefully make them more stable.)

All I know is I like having a lot of Checks (even if not required) that can actually prevent any and all exceptions to begin with.

And yes from what I learned from experiance myself. Always use try and catch anything before it is to late will help you in the long run.

(not to mention I blame Microsoft for not replacing their current Windows Update for when Updates Fail with a Detailed list of the names of the update(s) that failed and a description of what the reason is without Error code(s) as the Codes does not tell crap about why they failed. So, yeah I point my fingers at whoever wrote Windows Update, to fix their crap. As this is a issue in windows 7 SP1, 8, 8.1, and 10.)

Decorator: In python code you generally want to let the exception bubble up so that you can see what the error was. Sometimes you do want to catch general errors, but mostly you don't.

As for C++ try/catch, CPython is written in C, not C++.

So, your comment doesn't really have any relevance to this issue, I'm afraid.

I can confirm that I also get the same crash.

I do know for sure the crash happens in python.exe and blames ntdll.dll for it.

To Decorater
If you are found to have crashes, I'm sorry, you can follow the previous suggestion to use python_d.exe then open a new issue uploading crash dump.
It would be better.

About the multiprocessing question，I would like to add some information.

My code uses a multiprocessing Pool，I used multiprocessing.util.debug opened the debug information.debug information is as follows.

[DEBUG/SpawnPoolWorker-7] thread 'MainThread' does not own a connection
[DEBUG/SpawnPoolWorker-7] making connection to manager
[DEBUG/SyncManager-2] starting server thread to service 'SpawnPoolWorker-7'
[INFO/SpawnPoolWorker-4] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[INFO/SpawnPoolWorker-4] test:lock:handle: 712
[INFO/SpawnPoolWorker-7] test:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[INFO/SpawnPoolWorker-7] test:lock:handle: 768
[DEBUG/SpawnPoolWorker-7] DECREF 'e44c9bc720'
[INFO/SpawnPoolWorker-4] leave:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[DEBUG/SpawnPoolWorker-7] thread 'MainThread' has no more proxies so closing conn
[DEBUG/SpawnPoolWorker-7] DECREF 'b0182059f8'
[DEBUG/SyncManager-2] got EOF -- exiting thread serving 'SpawnPoolWorker-7'
[DEBUG/SpawnPoolWorker-4] INCREF 'e44c9bc720'
handle is closed
'ForkAwareLocal' object has no attribute 'connection'
[DEBUG/SpawnPoolWorker-7] thread 'MainThread' has no more proxies so closing conn
Traceback (most recent call last):
  File "C:/Users/Administrator/Desktop/Monitor_URL\Monitor_SQL.py", line 51, in execute_select_command
    conn.close()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 949, in __exit__
    return self._callmethod('release')
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\managers.py", line 716, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 204, in send
    self._check_closed()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\multiprocessing\connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed
[INFO/SpawnPoolWorker-7] Exception:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[INFO/SpawnPoolWorker-7] Exception process:lock:token: Token(typeid='Lock', address='\\\\.\\pipe\\pyc-3816-0-95051ql_', id='e44c9bc720')
[DEBUG/SyncManager-1] got EOF -- exiting thread serving 'SpawnPoolWorker-7'
[DEBUG/SpawnPoolWorker-4] INCREF 'b0182059f8'
[DEBUG/SpawnPoolWorker-4] DECREF 'b0182059f8'
[DEBUG/SpawnPoolWorker-7] INCREF 'e44c9bc720'

This time the message and not the same，From the information looks like two processes simultaneously locking the same lock object, and then release happened the problems.I do not know if this is correct，
But i will continue to follow the issue.

I use code that does multithreading to from a library that uses ffmpeg for exact it makes daemon threads so it could be from that.

I try to change the multiprocessing lib codes，upload the file is debugger display.I try to used while loop make sure connection hanle value is not none,but connection hanle value will still becomes none.
I think there may be other threads to modify the handle，But it not my code.

today，Find a new crash
python_d_crash_dump_20160816_A901A6F3.7z，is full crash dump
Download address
http://o8z0k9748.bkt.clouddn.com/python_d_crash_dump_20160816_A901A6F3.7z
dump display crash in the

python35_d.dll!visit_decref(_object * op, void * data) Line 379 C
python35_d.dll!dict_traverse(_object * op, int(*)(_object *, void *) visit, void * arg) Line 2534 C
python35_d.dll!subtract_refs(_gc_head * containers) Line 401 C
python35_d.dll!collect(int generation, __int64 * n_collected, __int64 * n_uncollectable, int nofail) Line 959 C

pythongh-71936: Fix race condition in multiprocessing.Pool

gh-71936: Fix race condition in multiprocessing.Pool (GH-124973)

Merged in main. I'll watch the buildbots for a bit before backporting.

@encukou

@encukou, sorry that it took me this long to reply but using the code in #71936 (comment) and your patch from #124973, I can no longer reproduce the problem!

Thanks a lot!

pythongh-71936: Fix race condition in multiprocessing.Pool (pythonGH-…

[3.12] gh-71936: Fix race condition in multiprocessing.Pool (GH-124973)…

[3.13] gh-71936: Fix race condition in multiprocessing.Pool (GH-124973)…

The backports are in. Thank you to everyone involved!

pythongh-71936: Fix race condition in multiprocessing.Pool (pythonGH-…

Link to upstream python bug python/cpython#71936

Navigation Menu

Repository navigation

multiprocessing Manager error: send() called for a closed connection #71936

Linked PRs

23 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

multiprocessing Manager error: send() called for a closed connection #71936

Description

Linked PRs

Activity

wevsty commented on Aug 12, 2016

eryksun commented on Aug 13, 2016

wevsty commented on Aug 13, 2016

AraHaan commented on Aug 13, 2016

bitdancer commented on Aug 13, 2016

AraHaan commented on Aug 13, 2016

AraHaan commented on Aug 13, 2016

wevsty commented on Aug 14, 2016

AraHaan commented on Aug 14, 2016

wevsty commented on Aug 15, 2016

wevsty commented on Aug 16, 2016

23 remaining items

encukou commented on Nov 13, 2024

basnijholt commented on Nov 15, 2024

encukou commented on Nov 18, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions