Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems #90876

Activity

1. maggyero commented on Feb 11, 2022

   

   maggyeromannequin

   on Feb 11, 2022

   MannequinAuthor

   More actions

   Any path-like object can be passed to multiprocessing.set_executable, i.e. objects with str, bytes, or os.PathLike type.

   For instance these work (tested on MacOS with all start methods: ‘spawn’, ‘fork’, and ‘forkserver’):

   • multiprocessing.set_executable(sys.executable) (str);
   • multiprocessing.set_executable(sys.executable.encode()) (bytes);
   • multiprocessing.set_executable(pathlib.Path(sys.executable)) (os.PathLike).

   This is because the ‘fork’ start method does not exec any program in the subprocess, the ‘spawn’ start method converts its path argument to bytes with os.fsencode before passing to _posixsubprocess.fork_exec, and the ‘forkserver’ start method spawns a server process (like with the ‘spawn’ start method) which then forks itself at each request (like the ‘fork’ start method):

           return _posixsubprocess.fork_exec(
               args, [os.fsencode(path)], True, passfds, None, None,
               -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
               False, False, None, None, None, -1, None)
   

   Linux (and other Unix-like systems) uses the same code than MacOS for the three start methods so it should work for it too.

   However I have not tested this on Windows which uses the function _winapi.CreateProcess for the ‘spawn’ start method (the only start method available on this OS) but I noticed that no conversion to str (not to bytes this time, since the function expects str) of the path argument with os.fsdecode (not os.fsencode this time) is performed before passing it to the function:

                   hp, ht, pid, tid = _winapi.CreateProcess(
                       python_exe, cmd,
                       None, None, False, 0, env, None, None)
   

   So on Windows only str path can be passed to multiprocessing.set_executable. This PR fixes this to be on a par with Unix-like systems which accept any path-like objects.
2. maggyero

   added

   3.9 (EOL)end of lifeend of life

   3.10 (EOL)end of lifeend of life

   3.11only security fixesonly security fixes

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-featureA feature request or enhancementA feature request or enhancement

   on Feb 11, 2022
3. maggyero

   changed the title [-]Add support of path-like objects to multiprocessing.set_executable for Windows to match Unix-like systems[/-] [+]Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems[/+] on Feb 11, 2022
4. maggyero

   changed the title [-]Add support of path-like objects to multiprocessing.set_executable for Windows to match Unix-like systems[/-] [+]Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems[/+] on Feb 11, 2022
5. ezio-melotti

   transferred this issue fromon Apr 10, 2022
6. brettcannon commented on Apr 22, 2022

   

   brettcannon

   on Apr 22, 2022

   Member

   More actions

   Closed via #31279 .
7. brettcannon

   closed this as completedon Apr 22, 2022
8. gpshead

   added 2 commits that reference this issue on Jul 5, 2023

   pythongh-90876: Prevent a multiprocessing import error.

   ...

   abb1037

   gh-90876: Restore the ability to import multiprocessing when `sys.exe…

   ...

   c60df36
9. miss-islington

   added 2 commits that reference this issue on Jul 6, 2023

   pythongh-90876: Restore the ability to import multiprocessing when `s…

   ...

   f17893b

   pythongh-90876: Restore the ability to import multiprocessing when `s…

   ...

   d28b815
10. gpshead

    added 2 commits that reference this issue on Jul 6, 2023

    [3.12] gh-90876: Restore the ability to import multiprocessing when `…

    ...

    4787eae

    [3.11] gh-90876: Restore the ability to import multiprocessing when `…

    ...

    1931c2a