Repository navigation
Spaces at the end of file and directory names are silently ignored on windows #115104
Description
Activity
This is normal Windows behaviour. Every Windows API that normalises paths (most of them, though most allow using the
\\?\prefix to bypass normalisation) will strip trailing dots and spaces from path segments.We won't make any change to support it by default, but we do try to make sure that if you use a
\\?\prefix then anything we might do with the path will work. But it's the caller's responsibility to get it right - we don't try to implement POSIX rules around Windows paths.Reacted by Zachary Ware, Serhiy Storchaka and bryceAt its most general, we pass path strings directly to Windows APIs. The best way to get Python to handle these in a particular way is to get Windows to handle them in a particular way.
Should the Using doc Windows section include the pair of sentences quoted above?
Yeah, somewhere we ought to write down our policy on this. I haven't figured out where best to put it though... seems kinda like Dev Guide is the place?
Yeah, somewhere we ought to write down our policy on this. I haven't figured out where best to put it though... seems kinda like Dev Guide is the place?
Isn't this rather user-facing? I think adding to https://docs.python.org/3/library/os.path.html somewhere in the vicinity of this note would be helpful:
Something like:
These modules mimic OS specific behavior (often by calling system APIs) and therefore may differ slightly in behavior across systems. For example, Windows typically trims trailing spaces from file and directory names.
The policy on how we choose what to control vs what to pass through is specific to contributors.
We should also document somewhere that, at least for Windows, we prefer to expose the OS behaviour "warts and all". I'm not sure exactly how we approach other platforms.
I also don't like listing specific behaviours, because they vary more often than you'd expect. Your "for example" is fine, but let's not try to make it an exhaustive list.
It probably also belongs in the os module docs, along with the other platform-specific notes near the top. (It's unfortunate that
os.pathalso accesses the filesystem and so suffers from OS-specific behaviours, but properly the relevant module isos.)Reacted by Tim Hoffmann- added 9 commits that reference this issue
on Jan 16, 2025 - added a commit that references this issue
on Jan 28, 2025 - added a commit that references this issue
on Mar 26, 2025 - added a commit that references this issue
on May 1, 2025 There's a real failure mode with the path normalization being inconsistent across
osfunctions.
As mentioned in the ticket,os.path.isdir()strips the trailing spaces.
Unfortunately,os.walk()doesn't, it returns an empty iterator by default. It creates a very non-intuitive case when a path to directory is validated as real, but the directory contents are silently skipped.
While either behavior is understandable in isolation, their inconsistency across a single package comes as user-unfriendly to me. A user may expect the uniform treatment of essentially the same argument. Basically,oslets its abstraction leak.As mentioned in the discussion, Windows is stripping trailing spaces, not us. It's entirely possible that we are doing something different with
os.walkto bypass OS normalisation, but unlikely - could you provide some sample code showing it?It's far more likely that a caller is bypassing it themselves by using a
\\?\-prefixed path.@zooba Sure:
import os import tempfile with tempfile.TemporaryDirectory() as d: p = os.path.join(d, "test") os.mkdir(p) q = p + " " print("q:", repr(q)) print("exists:", os.path.exists(q)) print("isdir:", os.path.isdir(q)) print("walk:", list(os.walk(q))) def onerror(e): print("walk error:", type(e).__name__, e, "filename:", getattr(e, "filename", None)) print("walk with onerror:") print(list(os.walk(q, onerror=onerror)))
Results:
q: 'C:\\Users\\am2\\AppData\\Local\\Temp\\tmpi_6cnybm\\test ' exists: True isdir: True walk: [] walk with onerror: walk error: FileNotFoundError [WinError 3] The system cannot find the path specified: 'C:\\Users\\am2\\AppData\\Local\\Temp\\tmpi_6cnybm\\test ' filename: C:\Users\am2\AppData\Local\Temp\tmpi_6cnybm\test []Both
os.path.exists()andos.path.isdir()say it's a thing, whileos.walk()silently (by default) returns an empty iterator.
Yes, it is avoidable once you know it, but it would be great if the default behavior would be not burning the users.Agreed, it looks like
FindFirstFileWdoesn't normalise individual directory segments, so if yount._findfirstfile("C:\\dir \\*")then it fails.That's ultimately not our fault, and I don't like when we write code to work around the OS design, but it also seems weird that this behaviour could've gone unnoticed for so long?
Can you confirm your version of Windows? The build number from the System page, ideally - I'm seeing it on my build, which is likely not out in public yet.
And if anyone else can test
nt._findfirstfile(nt.getcwd() + " \\*")and report the result and your OS version (especially if you haven't updated recently), that'd be helpful too.The OS trims whitespace from the end of a path. If you tried testing if
C:\\dir \\exists then that would fail too.but it also seems weird that this behaviour could've gone unnoticed for so long?
I would guess it's unusual for people to insert spaces at the end of paths.
I could've sworn it used to trim whitespace from each segment, not just the end, though it does still trim a trailing dot from each part of the path.
I think our policy of "as close to the OS behaviour as is portable" still applies here, but because we're the ones appending
\\*to the path forscandir, trimming spaces first isn't unreasonable. I'd rather use an API that doesn't involve us modifying the path, but if that doesn't exist, then we don't have a lot of choice.But since the original issue was opened for "what's the general policy on this" and that's been answered, let's start a new issue specifically for
scandir.@zooba Re: Windows version, my dev machine runs Win11 Pro 10.0.26200.
The machine where the issue occurred initially runs Server 2019 1809 (build 17763.8644).@ChrisDenton Sometimes, the paths come from e.g. Excel specifications where it's easy for the end users to Ctrl-V some gibberish, trailing spaces are not even the worst; the effort was made to validate the input, but this inconsistency between two functions really worked against my mental model.
Bug report
Bug description:
The explorer also shows the created directory as
test, i.e. without trailing spaces.The win32 docs say about trailing spaces
One could discuss whether this is user-responsibility, but silently dropping trailing spaces feels wrong. Either create the directory (though not recommended), or raise if there is a space at the end.
Note: Behaviour gets even more weird when combining trailing spaces with alternate data streams, e.g. "test : foo" (matplotlib/matplotlib#27748), which brought me here. But before going into this more difficult discussion, I'd like to learn the general position on handling trailings spaces in windows filenames.
CPython versions tested on:
3.12
Operating systems tested on:
Windows