resource tracker destroys shared memory segments when other processes should still have valid access

@pitrou

BPO	38119
Nosy	@pitrou, @applio, @pablogsal, @maggyero, @vinay0410, @JSmith-BitFlipper, @keven425, @TKaluza
PRs	bpo-38119: Fix shmem resource tracking #15989 gh-82300: Disable resource tracking & auto-destruction of SharedMemory blocks on POSIX #21516 bpo-38119: fix shared memory's resource tracking #23174
Files	mprt_monkeypatch.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/applio'
closed_at = None
created_at = <Date 2019-09-11.15:58:26.817>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'resource tracker destroys shared memory segments when other processes should still have valid access'
updated_at = <Date 2022-02-10.20:53:56.777>
user = 'https://github.com/applio'

bugs.python.org fields:

activity = <Date 2022-02-10.20:53:56.777>
actor = 'maggyero'
assignee = 'davin'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-09-11.15:58:26.817>
creator = 'davin'
dependencies = []
files = ['49859']
hgrepos = []
issue_num = 38119
keywords = ['patch']
message_count = 14.0
messages = ['351960', '352050', '374434', '374939', '374944', '374951', '374954', '374956', '374958', '374974', '387606', '388287', '390198', '412440']
nosy_count = 12.0
nosy_names = ['pitrou', 'steve.newcomb', 'turicas', 'davin', 'pablogsal', 'maggyero', 'vinay0410', 'damian.barabonkov', 'keven425', 'davfelsen', 'jdogzz-g5', 'timka']
pr_nums = ['15989', '21516', '23174']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38119'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

TODO list:

Add a track=True parameter to SharedMemory (gh-82300: Add track parameter to shared memory #110778 did this) so it can be disabled by the user.
It still needs covering in the What's New in 3.13 documentation.
Verify and make other documentation improvements as necessary as alluded by @ambv below in resource tracker destroys shared memory segments when other processes should still have valid access #82300 (comment) - It sounded like we've got docs implying that resources outlive the process by default which was not what the implementation ever did.

Linked PRs

gh-82300: Add track parameter to shared memory #110778

The resource tracker currently destroys (via _posixshmem.shm_unlink) shared memory segments on posix systems when any independently created Python process with a handle on a shared memory segment exits (gracefully or otherwise). This breaks the expected cross-platform behavior that a shared memory segment persists at least as long as any running process has a handle on that segment.

As described with an example scenario in bpo-37754:
Let's say a three processes P1, P2 and P3 are trying to communicate using shared memory.
--> P1 creates the shared memory block, and waits for P2 and P3 to access it.
--> P2 starts and attaches this shared memory segment, writes some data to it and exits.
--> Now in case of Unix, shm_unlink is called as soon as P2 exits. (This is by action of the resource tracker.)
--> Now, P3 starts and tries to attach the shared memory segment.
--> P3 will not be able to attach the shared memory segment in Unix, because shm_unlink has been called on that segment.
--> Whereas, P3 will be able to attach to the shared memory segment in Windows.

Another key scenario we expect to work but does not currently:

A multiprocessing.managers.SharedMemoryManager is instantiated and started in process A.
A shared memory segment is created using that manager in process A.
A serialized representation of that shared memory segment is deserialized in process B.
Process B does work with the shared memory segment that is also still visible to process A.
Process B exits cleanly.
Process A reads data from the shared memory segment after process B is gone. (This currently fails.)

The SharedMemoryManager provides a flexible means for ensuring cleanup of shared memory segments. The current resource tracker attempts to treat shared memory segments as equivalent to semaphore references, which is too narrow of an interpretation. As such, the current resource tracker should not be attempting to enforce cleanup of shared memory segments because it breaks expected behavior and significantly limits functionality.

Hi Davin,
This PR would fix the issues mentioned by you, by not prematurely unlinking the shared memory segment. And, therefore it would make shared memory useful in a lot of use cases.

But, this would still not make Unix's implementation consistent with Windows.
Windows uses a reference counting mechanism to count the number of processes using a shared memory segment. When all of them are done using it, Windows simply unlinks and frees the memory allocated to the shared memory segment.

I know that you already know this. I am commenting to find out, that what would be the next steps to fix the above inconsistency. You could see my last comment(msg351445) in bpo-37754, where I have listed some ways to implement the above reference counting mechanism.

If you could have a look and see which one would be the best way, I would be happy to make a PR for it.

@davin

@davin, could you merge one or the other of the PRs that fix this? Presumably also backport to 3.9 and 3.8 (but that's up to you and the release manager).

As per Guido's comment (#21516 (comment)), I'm going to use this space to discuss ways to go forward with resource tracking and SharedMemory.

Taking inspiration from Vinay (https://bugs.python.org/issue37754#msg351445), I think the simplest and best way forward is to use a small section of the shared memory at the start as a reference counter.

Every time a process latches onto a shared memory block, it does and atomic increment to the reference counter. And if it detaches, it does an atomic decrement. This atomic operations are available in C via hardware specific instructions. This would require modifying the Python C code posixshmem.c. It should not be a difficult change.

This would then change the SharedMemory API such that a call to close() could check the reference count at the end, and aromatically unlink if it reaches 0. Basically, the purpose of the explicit unlink() call is dissolved.

I think this would also play nice with the current implementation of the resource_tracker. A small change would need to take place such that it calls close() instead of unlink() as the clean up function. Nonetheless, it would keep track if all attachments of shared memory call close() at the end, which they should, and issue a warning if they do not. It would do this with the current code, no need to change anything.

I recommend bringing this new proposal up on python-dev or python-ideas, to get more eyeballs on the ideas before attempting implementation. One immediate worry I have is that if the reference counter is maintained in the shared memory segment, every process has to participate, and if a process crashes (segfaults) the shared refcount will be forever wrong and the segment will leak. Distributed GC is hard!

That's a valid point Guido. But, I guess this can be easily handled by resource tracker. At this current moment resource tracker unlinks shared memory if the process which created it dies without unliking it.

Therefore, resource tracker handles cleaning up resources which the respective processes couldn't or didn't do.

So, instead of unlinking the shared memory segment, resource tracker can instead decrement the reference count, if the process failed to do so, and if the reference count becomes 0, then unlink the shared memory segment.

This approach will ensure that even if the respective processes died unexpectedly, there are no leaks.

Unless the resource_tracker also dies along with the process. In which case, I'm not sure what there is there to do.

I believe the resource_tracker actually spawns a process alongside the process that uses it. So if the parent process seg-faults, the resource_tracker should still be alive.

Well, the chances of resource tracker dying abruptly are very less because it's thoroughly tested, and there are mechanisms to re-spawn resource_tracker process if you see the code. There is a function called def ensure_running.

Resource tracker is still alive even if the process for which it was created dies. It also handles cleaning shared semaphores. So, I guess this is something we can rely on for cleaning up things, because at the end of the day that's what it was made for.

Agreed.

Thanks for all the discussion & proposed PRs everybody.

@pan324

@pan324 Thanks for continuing to bang on the PR until it got through.

@gpshead Thanks for shepherding this in.

Thanks Greg. I was out of my depth here…

pythongh-82300: Add track parameter to multiprocessing.shared_memory (p…

Backport: gh-82300: Add track parameter to multiprocessing.shared_memory

Glad to see this get resolved. For a backward-compatible fix, I have found that the below works well in case anybody needs this in earlier Python versions:

import sys
import threading
from multiprocessing import resource_tracker as _mprt
from multiprocessing import shared_memory as _mpshm


if sys.version_info >= (3, 13):
    SharedMemory = _mpshm.SharedMemory
else:
    class SharedMemory(_mpshm.SharedMemory):
        __lock = threading.Lock()

        def __init__(
            self, name: str | None = None, create: bool = False,
            size: int = 0, *, track: bool = True
        ) -> None:
            self._track = track

            # if tracking, normal init will suffice
            if track:
                return super().__init__(name=name, create=create, size=size)

            # lock so that other threads don't attempt to use the
            # register function during this time
            with self.__lock:
                # temporarily disable registration during initialization
                orig_register = _mprt.register
                _mprt.register = self.__tmp_register

                # initialize; ensure original register function is
                # re-instated
                try:
                    super().__init__(name=name, create=create, size=size)
                finally:
                    _mprt.register = orig_register

        @staticmethod
        def __tmp_register(*args, **kwargs) -> None:
            return

        def unlink(self) -> None:
            if _mpshm._USE_POSIX and self._name:
                _mpshm._posixshmem.shm_unlink(self._name)
                if self._track:
                    _mprt.unregister(self._name, "shared_memory")

This is likely slightly less efficient than simply copying the 3.13 version of SharedMemory, but is a lot smaller (especially given only a few things need to happen in order for tracking to be avoided).

pythongh-82300: Add track parameter to multiprocessing.shared_memory (p…

fix a Python SharedMemory free Bug

ENH: Grid optimization with mp.Pool & mp.shm.SharedMemory

my application is using ShareableList() which makes use of SharedMemory() => shared_memory.py .. actually I was glad to see that the track parameter got added to SharedMemory() as of 3.13 onwwards, but I was wondering why this track parameter was not added to ShareableList() also ... doing this will help standalone client processes to close without destroying the ShareableList() of the server process.
Below you see the necessary adaptations in the code for this:

`class ShareableList:

def __init__(self, sequence=None, *, name=None, track=True):

    if name is None or sequence is not None:

        self.shm = SharedMemory(name, create=True, size=requested_size, track=track)
    else:
        self.shm = SharedMemory(name, track=track)

`

Python < 3.13 tracked by default, I thought it didn't track!

ENH: Grid optimization with mp.Pool & mp.shm.SharedMemory

Navigation Menu

Repository navigation

resource tracker destroys shared memory segments when other processes should still have valid access #82300

TODO list:

Linked PRs

32 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

resource tracker destroys shared memory segments when other processes should still have valid access #82300

Description

TODO list:

Linked PRs

Activity

applio commented on Sep 11, 2019

vinay0410 commented on Sep 12, 2019

gvanrossum commented on Jul 27, 2020

JSmith-BitFlipper commented on Aug 6, 2020

gvanrossum commented on Aug 6, 2020

vinay0410 commented on Aug 6, 2020

JSmith-BitFlipper commented on Aug 6, 2020

vinay0410 commented on Aug 6, 2020

JSmith-BitFlipper commented on Aug 6, 2020

32 remaining items

gpshead commented on Dec 5, 2023

buzmeg commented on Dec 11, 2023

gvanrossum commented on Dec 11, 2023

tanrbobanr commented on Jun 15, 2024

mbrinz commented on Mar 2, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions