crc32 function outputs wrong result for large data on the macOS arm64 platform

[maikschulze]

Bug report

The functions zlib.crc32 and binascii.crc32 share the problematic behavior. When computing the CRC for data >= 2GB macOS arm64 binaries result in different values than all other platforms such as macOS x64, Windows x64, Linux x64. Consequently, problems arise e.g. when using the zipfile module.

A clear and concise description of what the bug is.

Reproduction:

import random
random.seed(0)

import zlib
import binascii

def chunks(list, n):
  for i in range(0, len(list), n):
      yield list[i:i + n]

random_megabyte = random.randbytes(1024*1024)
random_1_gigabyte = random_megabyte * 1024 * 1
random_4_gigabyte = random_megabyte * 1024 * 4

crc_1_gigabyte_zlib = zlib.crc32(random_1_gigabyte, 0)
crc_1_gigabyte_binascii = binascii.crc32(random_1_gigabyte, 0)

crc_4_gigabyte_zlib = zlib.crc32(random_4_gigabyte, 0)
crc_4_gigabyte_binascii = binascii.crc32(random_4_gigabyte, 0)

# incremental computation in chunks < 2 GB fixes macOS arm64
chunked_crc_4_gigabyte_zlib = 0
chunked_crc_4_gigabyte_binascii = 0
for chunk in chunks(random_4_gigabyte, 1024 * 1024 * 1024 *1):
  chunked_crc_4_gigabyte_zlib = zlib.crc32(chunk, chunked_crc_4_gigabyte_zlib)
  chunked_crc_4_gigabyte_binascii = binascii.crc32(chunk, chunked_crc_4_gigabyte_binascii)

print("crc_1_gigabyte_zlib".ljust(32),             "expected: 0xe28bc234 computed:", hex(crc_1_gigabyte_zlib))
print("crc_1_gigabyte_binascii".ljust(32),         "expected: 0xe28bc234 computed:", hex(crc_1_gigabyte_binascii))
print("crc_4_gigabyte_zlib".ljust(32),             "expected: 0x278432d6 computed:", hex(crc_4_gigabyte_zlib))
print("crc_4_gigabyte_binascii".ljust(32),         "expected: 0x278432d6 computed:", hex(crc_4_gigabyte_binascii))
print("chunked_crc_4_gigabyte_zlib".ljust(32),     "expected: 0x278432d6 computed:", hex(chunked_crc_4_gigabyte_zlib))
print("chunked_crc_4_gigabyte_binascii".ljust(32), "expected: 0x278432d6 computed:", hex(chunked_crc_4_gigabyte_binascii))

Output on macOS arm64:

mac-arm64:crc_bug dev_admin$ /opt/homebrew/bin/python3 crc_bug_report.py 
crc_1_gigabyte_zlib              expected: 0xe28bc234 computed: 0xe28bc234
crc_1_gigabyte_binascii          expected: 0xe28bc234 computed: 0xe28bc234
crc_4_gigabyte_zlib              expected: 0x278432d6 computed: 0x6b54c6be
crc_4_gigabyte_binascii          expected: 0x278432d6 computed: 0x6b54c6be
chunked_crc_4_gigabyte_zlib      expected: 0x278432d6 computed: 0x278432d6
chunked_crc_4_gigabyte_binascii  expected: 0x278432d6 computed: 0x278432d6

Your environment

• CPython versions tested on: Python 3.9.6 Python 3.11.4
• Operating system and architecture: macOS arm64, macOS x64, Windows x64, Linux x64

Linked PRs

• gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig #112615
• [3.12] gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig (GH-112615) #112724
• [3.11] gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig (GH-112615) #112725

[ned-deily]

Thanks for the report. The Python standard library zlib and binascii modules depend on the widely-used third-party zlib library (libz) to do much of their work. The misbehavior shown in your test case appears to be due to a bug of some sort in the native Apple Silicon (arm64) libz binary shipped by Apple with current releases of macOS. The incorrect results can be seen on Apple Silicon Macs running any current release of macOS that support arm64, that is, macOS 11 Big Sur, macOS 12 Monterey, and macOS 13 Ventura. The upcoming macOS 14 Sonoma, which is currently in restricted beta, appears likely to be released with an updated version of libz that does not have this problem.

The Apple-supplied libz on macOS 11 through 13 reports its version as 1.2.11. However, a quick look at the zlib project files published on Apple's open-source page (https://opensource.apple.com/releases/) shows that the Apple version has been significantly patched from the original upstream 1.2.11 release (linked to from https://www.zlib.net). I was unable to reproduce the failure here on either Apple Silicon or Intel Macs when using a libz built from unmodified upstream zlib source for 1.2.11, 1.2.12, or 1.2.13 (the most recent release) nor with current libz 1.2.13 binaries from either Homebrew or MacPorts. So, the problem seems to be confined to the Apple-supplied libz on Apple Silicon Macs.

By default, when building on macOS, cpython will dynamically link with a number of open-source libraries supplied with macOS, including libz. While there are advantages to this approach and has been encouraged by Apple in the past, it can be a disadvantage if the system-supplied version has a bug.

A non-exhaustive survey of several ways of using cpython on macOS:

1. pythons provided by all current python.org macOS universal2 installers dynamically link to the Apple-supplied system libz and thus exhibit this problem. (A vary quick and rather dirty mitigation for python.org universal2 pythons is to force the use of Intel emulation on Apple Silicon Macs by installing Rosetta2 and using python3.11-intel64 instead of python3.11 to invoke the interpreter, all at the risk of degraded Python performance and other new bugs.)

2. pythons provided by Homebrew also link to the system libz and exhibit this problem (spot checked with Homebrew's current Python 3.9 through 3.11).

3. pythons provided by MacPorts dynamically link with a MacPorts-provided libz and thus do not exhibit this problem.

4. By default, pythons built from source will link with the system libz and exhibit this problem. Linking with a non-system libz can be specified in the ./configure build step by various means including the setting of CFLAGS, CPPFLAGS, and LDFLAGS environment variables or, as of 3.11, with pkg-config if available.

This is not the first time that problems have been found using the zlib and binascii modules with crc32 and very large data buffers, for example, #54485 and, more recently, #82437. (The Python 3.9.6 supplied with current releases of Apple's developer tools exhibits both the problem identified in this issue and the problem in #82437 which was fixed in a later release of 3.9.)

So, assuming the above analysis is correct and that there is no problem within cpython itself, what should be done to mitigate this? Ideally, Apple would address the problem with an updated libz while all three affected macOS release families are still receiving updates. To that end, we should then ensure that an issue is open with Apple about this problem. Given the uncertainty of if and when such a fix would be available, distributors of cpython for macOS and users who build cpython from source will need to evaluate the risk to their users and applications. If necessary, builds can be changed to use a non-system libz. That would be fairly straight-forward for distributions like the python.org installers and Homebrew which already make use of third-party dynamic shared libraries. Stand-alone applications need to be careful when linking and shipping a new shared library, perhaps needing RPATH directives, or by forcing static linking with libz, something which macOS deliberately does not make easy to do.

Comments? Other options?

cc: @ronaldoussoren, @ambv, @gpshead. @Yhg1s

[ronaldoussoren]

We should definitely file and issue with Apple about this, ideally with a reproducer in C. A fix in the macOS 14 beta doesn't bode well for getting this fixed in older releases though. That said, most fixes I've seen for issues I filed about system libraries like this were fixed in new major releases anyway.

Other than that I'm not sure. Do you know what kind of changes there are in the Apple for of libz and if those are relevant for our users (for example by enabling some kind of hardware offload)?

If the changes aren't relevant we could start shipping our own copy of libz, the build tooling is set up for that although it would add yet another moving part to track and be concerned about when there's a release that doesn't fit into the cpython release schedule.

Another option we could look into: Is it possible to tweak the our CRC32 bindings to call the libz function with small enough buffers to not trigger the issue? The disadvantage of that it that it would complicate our code base, but hopefully not too much.

[gpshead]

@ned-deily libz is used by a huge pile of software other than Python, what do other things use instead of the system libz? While we could ship our own zlib on macOS but it'd be a shame as the standard zlib is very boring and unoptimized; dynamically linking against the platform one means they're responsible for security and performance updates.

[gpshead]

ex: zip and unzip on macOS, what do those use? what about Java and Ruby?

[gpshead]

From a risk perspective we already build our own upstream zlib 1.2.13 for Windows binary releases so statically linking against a build of that on macOS isn't a huge burden, zlib security fixes trigger Python binary security releases anyways.

#91349 (use zlib-chromium or zlib-ng) and #93819 (we need zlib in WASI) are tangentially related FWIW.

[gpshead]

re: Apple bug vs our bug, we should verify that by writing a tiny C reproducer using libz to confirm the issue.

From a CPython bugfix perspective, we do still carry a copy of our own crc32 function within the binascii module, conditionally compiled in or out (based on zlib) - the condition in which that is used could be adjusted as a platform workaround. (but this still sounds like there may be other internal Apple libz issues with visible impact?)

[arhadthedev]

we do still carry a copy of our own crc32 function within the binascii module, conditionally compiled in or out (based on zlib)

It will be eliminated by gh-32043 (see USE_ZLIB_CRC32 removals in Modules/binascii.c).

[markokr]

If there is wish to continue to link with platform libz, how about setting the loop limit to 1G instead 4G-1 (UINT32_MAX)? That would isolate the code from platform corner-cases and its unlikely to give any measurable slowdown.

[ronaldoussoren]

The following is a C program that tries to reproduce the issue based on the code in binascii.c:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "zlib.h"


int main(void)
{
    unsigned char* buffer = malloc(4L * 1024 * 1024 * 1024);
    if (buffer == NULL) {
        perror("malloc");
        return 2;
    }

    for (size_t k = 0; k < 4L * 1024 * 1024 * 1024; k++) {
        buffer[k] = k & 0xff;
    }

    unsigned int crc_whole = 0;
    unsigned int crc_chunked = 0;

    // use two chunks for crc_whole because 4G is larger than UINT_MAX,
    // 1 big chunk and 1 chunk for the rest
    crc_whole = crc32(crc_whole, buffer, 4L * 1024 * 1024 * 1024 - 1);
    crc_whole = crc32(crc_whole, buffer + 4L * 1024 * 1024 * 1024 - 1, 1);

    for (unsigned int i = 0; i < 4; i++) {
        crc_chunked = crc32(crc_chunked, buffer + (i * 1024 * 1024 * 1024), 1 * 1024 * 1024 * 1024);
    }

    printf("W: %#x\n", crc_whole);
    printf("C: %#x\n", crc_chunked);
    printf("W == C: %d\n", crc_whole == crc_chunked);

    return !(crc_whole == crc_chunked);
}

Build using cc -o repro repro.c -lz.

For me this prints:

% ./repro                                                                                                                                       (3.12)cpython-upstream
W: 0xe3bbf7be
C: 0xe3bbf7be
W == C: 1

That is, both approaches get the same CRC.

System version (M1 MacBook):

% sw_vers                                                                                                                                       (3.12)cpython-upstream
ProductName:		macOS
ProductVersion:		14.1.2
BuildVersion:		23B92

I get the same result in a VM running 10.15.5.

[ronaldoussoren]

My results are expected given @ned-deily's earlier remarks...

Limiting chunk size in calls to crc32(3) to 1 GB is the easiest workaround on our end.