Repository navigation
zipimport fails when the ZIP archive contains more than 65535 files #77140
Description
| BPO | 32959 |
|---|---|
| Nosy | @Yhg1s, @serhiy-storchaka, @bbayles |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2018-02-26.21:08:51.426>
labels = ['type-bug', 'library']
title = 'zipimport fails when the ZIP archive contains more than 65535 files'
updated_at = <Date 2018-09-19.07:19:25.103>
user = 'https://bugs.python.org/mchouza'bugs.python.org fields:
activity = <Date 2018-09-19.07:19:25.103>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-02-26.21:08:51.426>
creator = 'mchouza'
dependencies = []
files = []
hgrepos = []
issue_num = 32959
keywords = []
message_count = 3.0
messages = ['312957', '313044', '325718']
nosy_count = 4.0
nosy_names = ['twouters', 'mchouza', 'serhiy.storchaka', 'bbayles']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32959'
versions = ['Python 3.6']
Activity
mchouza commentedon Feb 26, 2018
When trying to import a module from a ZIP archive containing more than 65535 files, the import process fails:
$ python3 -VV Python 3.6.4 (default, Jan 6 2018, 11:49:38) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] $ cat create_zips.py from zipfile import ZipFile with ZipFile('below.zip', 'w') as zfp: for i in range(65535): zfp.writestr('m%d.py' % i, '') with ZipFile('over.zip', 'w') as zfp: for i in range(65536): zfp.writestr('m%d.py' % i, '') $ python3 create_zips.py $ python -m zipfile -l below.zip | (head -2 && tail -2) File Name Modified Size m0.py 2018-02-26 20:57:32 0 m65533.py 2018-02-26 20:57:36 0 m65534.py 2018-02-26 20:57:36 0 $ python -m zipfile -l over.zip | (head -2 && tail -2) File Name Modified Size m0.py 2018-02-26 20:57:36 0 m65534.py 2018-02-26 20:57:40 0 m65535.py 2018-02-26 20:57:40 0 $ PYTHONPATH=below.zip python3 -c 'import m0' $ PYTHONPATH=over.zip python3 -c 'import m0' Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'm0'I think the problem is related to the zipimport module not handling the 'zip64 end of central directory record'.
Yhg1s commentedon Feb 28, 2018
FWIW, yes, this is because zipimport doesn't support ZIP64, and doesn't even flag it as an error when the ZIP requires it. Instead it skips files; the ZIP64 format works by setting the fields that would overflow to the maximum value as a signal that the real value should be read from the ZIP64 records, which means it's still a valid non-64-bit ZIP file, just a truncated one.
Adding ZIP64 support to zipimport is not trivial, and requires breaking some behaviour that's bad (not according to the ZIP spec) but actually covered by zipimport's tests: prepending arbitrary data to the ZIP file by just concatenating them together (rather than rewriting the ZIP archive's offsets). I don't think there's going to be an easy fix here, although I am working on open-sourcing a replacement zipimport that I wrote for work, which does support ZIP64 properly (and also fixes some other fundamental flaws in zipimport). I hope to have that done soon, and we can discuss replacing zipimport with that module in Python 3.8 or later.
serhiy-storchaka commentedon Sep 19, 2018
zipimport have been rewritten in pure Python (bpo-25711).
gh-89739: gh-77140: Support zip64 in zipimport (GH-94146)
gpshead commentedon Mar 28, 2024
This feature should appear in the next 3.13 pre-release.
Squashed commit of the following:
Squashed commit of the following:
pythongh-89739: pythongh-77140: Support zip64 in zipimport (pythonGH-…