os.dup2(fd, fd, inheritable=False) behaves inconsistently

@benjaminp

BPO	32862
Nosy	@benjaminp, @eryksun, @izbyshev
PRs	gh-77043: Make os.dup2(fd, fd) a no-op for valid fd #5713

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-02-17.03:24:02.305>
labels = ['type-bug', '3.8', '3.9', '3.10', 'extension-modules', 'library']
title = 'os.dup2(fd, fd, inheritable=False) behaves inconsistently'
updated_at = <Date 2021-03-20.14:32:55.054>
user = 'https://github.com/izbyshev'

bugs.python.org fields:

activity = <Date 2021-03-20.14:32:55.054>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2018-02-17.03:24:02.305>
creator = 'izbyshev'
dependencies = []
files = []
hgrepos = []
issue_num = 32862
keywords = ['patch']
message_count = 3.0
messages = ['312266', '312295', '312298']
nosy_count = 3.0
nosy_names = ['benjamin.peterson', 'eryksun', 'izbyshev']
pr_nums = ['5713']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32862'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

gh-77043: Make os.dup2(fd, fd, False) always remove inheritability #102148

os.dup2(fd, fd, inheritable=False) may fail or change fd inheritability in following ways:

POSIX without F_DUP2FD_CLOEXEC
1.1) dup3() is available (a common case for Linux): OSError (EINVAL, dup3() doesn't allow equal descriptors)
1.2) dup3() is not available: fd made non-inheritable
POSIX with F_DUP2FD_CLOEXEC (FreeBSD): inheritability is not changed
Windows: fd made non-inheritable

In contrast, os.dup2(fd, fd, inheritable=True) never changes fd inheritability (same as before PEP-446 landed). I suggest to make os.dup2(fd, fd, inheritable=False) behave the same.

In Windows the CRT file descriptor is actually still inheritable. This only makes the underlying OS handle non-inheritable. I don't think there's a way to make an existing FD non-inheritable using public CRT functions. See bpo-32865.

@eryksun

@eryksun: Thank you for the note! I've commented on bpo-32865. This adds even more inconsistency to this corner case.

pythongh-77043: Make os.dup2(fd, fd, False) always remove inheritability

@vstinner

I've revisited this issue, and here is corrected and expanded list of how os.dup2(fd, fd, inheritable=False) behaves across platforms:

On Windows, Linux before 2.6.27, and macOS dup2() is attempted
(validating the fd), and on success the fd is then separately made
non-inheritable.
On platforms where F_DUP2FD_CLOEXEC fcntl command is used:
- The FreeBSD man page doesn't explicitly document behavior for the case of equal fds. The implementation does set FD_CLOEXEC in this case (I've also confirmed it in a VM with FreeBSD 12.2).
- The Solaris man page says this about F_DUP2FD_CLOEXEC: "Similar to F_DUP2FD, except that the FD_CLOEXEC flag associated with the new file descriptor is set, unless fildes is equal to arg, in which case the flags are unchanged." I have no way to test this.
- The Illumos man page says this about F_DUP2FD_CLOEXEC: "...If filedes equals arg, the call will fail setting errno to EINVAL." (essentially as dup3() on Linux). The source code confirms this.
On Linux 2.6.27+ dup3() is used, which always fails with EINVAL (even
if the fd is invalid).

So overall we have the full range of possible behavior across platforms: inheritability is changed, not changed, or OSError is raised.

One way to fix this inconsistency would be to make os.dup2(fd, fd, False) behave as os.dup2(fd, fd): just validate the fd and don't change its inheritability. This is what I originally attempted in PR #5713. But now, given that on all platforms except Solaris os.dup2(fd, fd, False) either fails or makes the fd non-inheritable, I think this change could be viewed as a regression: we probably don't want to introduce new cases where inheritable fds can unexpectedly appear, and the caller explicitly asks for inheritable=False here.

Likewise, changing os.dup2(fd, fd, False) to always fail (like dup3()) would add a new error case on several platforms and can be viewed as a regression.

If we were designing os.dup2() from scratch, the right thing to do would probably be to make inheritable always affect the inheritability (and make it False by default). But now we can't change how os.dup2() behaves by default (even PEP-446 didn't touch it).

I'm not sure whether it's worth to separate the default (inheritable is not passed explicitly) and the explicit inheritable=True case and make the latter always make the returned fd inheritable. But I think that something needs to be done to at least fix the inheritable=False mess. So I've implemented PR #102148 to always make the fd non-inheritable in this case.

Cc: @vstinner

pythongh-77043: Make os.dup2(fd, fd, False) always remove inheritability

As an user, I expect that os.dup2(fd, fd, inheritable=False) raises an exception if fd is invalid, or makes the file descriptor non-inheritable.

In Python in general, we attempt to have a consistent behavior on all platforms, adding platform-specific code for that if needed. For example, there is a lot of (platform-specific) code to have a portable behavior with Not-a-Number (NaN) in math functions.

For the special case fd2 == fd, we should avoid dup3(). Maybe just have the same portable code on all platforms for this special case.

As an user, I expect that os.dup2(fd, fd, inheritable=False) raises an exception if fd is invalid, or makes the file descriptor non-inheritable.

For the special case fd2 == fd, we should avoid dup3()

PR #102148 is in line with this.

Maybe just have the same portable code on all platforms for this special case.

I'm not aware of a portable way of validating an fd that is better than simply delegating to dup2() (like in PR #102148). The obvious way to validate an fd (fcntl) doesn't exist on Windows. Moreover, fd validation is not as easy as it might sound (see is_valid_fd), and an fd might be valid for some operations, but not for others. I don't think CPython should invent its own semantics of the fd validity for os.dup2() , risking it be different from what the actual dup2() does for the sake of providing a consistent behavior. The os module is low-level and has always been tied to the underlying libc, and users came to expect that. This situation is different from math functions.

Navigation Menu

Repository navigation

os.dup2(fd, fd, inheritable=False) behaves inconsistently #77043

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

os.dup2(fd, fd, inheritable=False) behaves inconsistently #77043

Description

Linked PRs

Activity

izbyshev commented on Feb 17, 2018

eryksun commented on Feb 18, 2018

izbyshev commented on Feb 18, 2018

izbyshev commented on Feb 22, 2023

vstinner commented on Apr 5, 2023

izbyshev commented on Apr 5, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions