Skip to content

--disable-gil & --enable-experimental-jit fails to build on M1 Mac #118844

Closed
Closed
--disable-gil & --enable-experimental-jit fails to build on M1 Mac#118844
Assignees
savannahostrowski
Labels
OS-macbuildThe build process and cross-buildtopic-JITtype-bugAn unexpected behavior, bug, or error

Description

@smontanaro
smontanaro
opened on May 9, 2024

Bug report

Bug description:

On my M1 Mac, if I configure like so: ./configure --disable-gil --enable-experimental-jit I get a compilation error:

gcc -c -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall    -D_Py_TIER2=1 -D_Py_JIT -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal -I./Include/internal/mimalloc  -I. -I./Include    -DPy_BUILD_CORE -o Python/jit.o Python/jit.c
In file included from Python/jit.c:392:
./jit_stencils.h:13419:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16a(code + 0x198, (uintptr_t)&_PyCriticalSection_Resume);

(followed by many more of the same error).

I don't recall if I've tried this configure combination before or not. I'm using Apple's clang on an up-to-date MacOS (14.4.1):

% gcc --version
Apple clang version 15.0.0 (clang-1500.3.9.4)
Target: arm64-apple-darwin23.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

Activity

  1. smontanaro
    added
    type-bugAn unexpected behavior, bug, or error
    on May 9, 2024
  2. Eclips4
    added
    OS-mac
    buildThe build process and cross-build
    on May 9, 2024
  3. Eclips4
    changed the title [-]--disable-gil & --wirh-experimental-jit fails to build on M1 Mac[/-] [+]--disable-gil & --enable-experimental-jit fails to build on M1 Mac[/+] on May 9, 2024

  4. Eclips4 commented on May 9, 2024

    @Eclips4
    Eclips4
    on May 9, 2024
    Member

    Full compilation error:

    Details 
    gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -D_Py_TIER2=1 -D_Py_JIT -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal -I./Include/internal/mimalloc  -I. -I./Include    -DPy_BUILD_CORE -o Python/jit.o Python/jit.c
In file included from Python/jit.c:392:
./jit_stencils.h:24576:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16a(code + 0x230, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:24577:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16b(code + 0x234, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:24578:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16c(code + 0x238, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:24579:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16d(code + 0x23c, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:24596:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16a(code + 0x2a8, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:24597:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16b(code + 0x2ac, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:24598:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16c(code + 0x2b0, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:24599:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16d(code + 0x2b4, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:45384:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16a(code + 0x1c0, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:45385:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16b(code + 0x1c4, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:45386:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16c(code + 0x1c8, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:45387:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16d(code + 0x1cc, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:45404:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16a(code + 0x238, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:45405:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16b(code + 0x23c, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:45406:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16c(code + 0x240, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:45407:49: error: use of undeclared identifier '_PyCriticalSection_BeginSlow'
    patch_aarch64_16d(code + 0x244, (uintptr_t)&_PyCriticalSection_BeginSlow);
                                                ^
./jit_stencils.h:48060:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16a(code + 0x358, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:48061:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16b(code + 0x35c, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
./jit_stencils.h:48062:49: error: use of undeclared identifier '_PyCriticalSection_Resume'
    patch_aarch64_16c(code + 0x360, (uintptr_t)&_PyCriticalSection_Resume);
                                                ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [Python/jit.o] Error 1

  5. brandtbucher commented on May 9, 2024

    @brandtbucher
    brandtbucher
    on May 9, 2024
    Member

    Thanks! I’m not sure if this is a build configuration that’s been given any thought yet... or if it even makes sense right now (I believe our tier 2 optimization pipeline is disabled on free-threaded builds, but I’d need to check). Regardless, we’re headed towards a future with both of these features and should probably make sure that they at least build together.

    @savannahostrowski, interested in taking this too? I’m on my phone now, but I think we probably just need to #include the header where this is defined in jit.c.

  6. brandtbucher
    self-assigned this
    on May 9, 2024

  7. sobolevn commented on May 9, 2024

    @sobolevn
    sobolevn
    on May 9, 2024
    Member

    Here' the code sample that is generated:

    memcpy(data, data_body, sizeof(data_body));
    patch_64(data + 0x0, instruction->oparg);
    memcpy(code, code_body, sizeof(code_body));
    patch_aarch64_33rx(code + 0x30, (uintptr_t)data);
    patch_aarch64_26r(code + 0x16c, (uintptr_t)code + sizeof(code_body));
    patch_aarch64_33rx(code + 0x190, (uintptr_t)data);
    patch_aarch64_26r(code + 0x1c8, instruction_starts[instruction->error_target]);
    patch_aarch64_16a(code + 0x1d0, (uintptr_t)&_PyEval_FormatExcUnbound);
    patch_aarch64_16b(code + 0x1d4, (uintptr_t)&_PyEval_FormatExcUnbound);
    patch_aarch64_16c(code + 0x1d8, (uintptr_t)&_PyEval_FormatExcUnbound);
    patch_aarch64_16d(code + 0x1dc, (uintptr_t)&_PyEval_FormatExcUnbound);
    patch_aarch64_16a(code + 0x1e8, (uintptr_t)&_PyCriticalSection_Resume);
    patch_aarch64_16b(code + 0x1ec, (uintptr_t)&_PyCriticalSection_Resume);
    patch_aarch64_16c(code + 0x1f0, (uintptr_t)&_PyCriticalSection_Resume);
    patch_aarch64_16d(code + 0x1f4, (uintptr_t)&_PyCriticalSection_Resume);
    patch_aarch64_16a(code + 0x200, (uintptr_t)&_PyThreadState_GetCurrent);
    patch_aarch64_16b(code + 0x204, (uintptr_t)&_PyThreadState_GetCurrent);
    patch_aarch64_16c(code + 0x208, (uintptr_t)&_PyThreadState_GetCurrent);
    patch_aarch64_16d(code + 0x20c, (uintptr_t)&_PyThreadState_GetCurrent);

  8. savannahostrowski commented on May 9, 2024

    @savannahostrowski
    savannahostrowski
    on May 9, 2024
    Member

    Yep - feel free to assign to me @brandtbucher! I'll take a look tonight or tomorrow!

  9. brandtbucher
    assigned
    savannahostrowski
    and unassigned
    brandtbucher
    on May 9, 2024

  10. smontanaro commented on May 9, 2024

    @smontanaro
    smontanaro
    on May 9, 2024
    ContributorAuthor

    FWIW, I'm not sure this is M1-specific. It compiled just fine on my Dell laptop running XUbuntu 22.04. OTOH, on an old MacBook Air I have laying around (dual core Intel i5, MacOS 12.7.4), it failed to compile Python/jit.c in the same fashion as on my M1 MBP. Maybe it's a MacOS issue?

  11. brandtbucher
    added a commit that references this issue on May 11, 2024

    GH-118844: Fix build failures when combining --disable-gil with --ena…

    5b941e5
  12. miss-islington
    added a commit that references this issue on May 11, 2024

    pythonGH-118844: Fix build failures when combining --disable-gil with…

    f3612d8
  13. brandtbucher
    closed this as completedon May 11, 2024

  14. smontanaro commented on May 13, 2024

    @smontanaro
    smontanaro
    on May 13, 2024
    ContributorAuthor

    @brandtbucher @savannahostrowski

    I saw the close-out of the PR and apparent backport to 3.13. I can build fine on main, but compilation still fails for me on 3.13. On both main and 3.13 I used this configure command (grepped from config.status):

    /bin/sh './configure'  '--enable-experimental-jit=yes' '--disable-gil' '--with-openssl=/opt/homebrew' 'GDBM_CFLAGS=-I/opt/homebrew/opt/gdbm/include' 'GDBM_LIBS=-L/opt/homebrew/opt/gdbm/lib -lgdbm' 'LIBLZMA_CFLAGS=-I/opt/homebrew/include' 'LIBLZMA_LIBS=-L/opt/homebrew/lib -llzma' $ac_configure_extra_args --no-create --no-recursion

    I thought the PR was backported to 3.13, but maybe not? Diffing the main and 3.13 versions of Python/jit.c I see this line is missing from the 3.13 version:

    #include "pycore_critical_section.h"

    Adding it allows the compilation to succeed. I don't see it in the jit.c source on Github either.

  15. brandtbucher commented on May 13, 2024

    @brandtbucher
    brandtbucher
    on May 13, 2024
    Member

    I saw the close-out of the PR and apparent backport to 3.13. I can build fine on main, but compilation still fails for me on 3.13.

    Ah, sorry, looks like the backport failed! Re-running it now...

  16. brandtbucher
    added a commit that references this issue on May 13, 2024

    [3.13] GH-118844: Fix build failures when combining --disable-gil wit…

    9d2c10b

  17. brandtbucher commented on May 13, 2024

    @brandtbucher
    brandtbucher
    on May 13, 2024
    Member

    ...done!

  18. brandtbucher
    added
    topic-JIT
    on May 31, 2024
  19. estyxx
    added a commit that references this issue on Jul 17, 2024

    pythonGH-118844: Fix build failures when combining --disable-gil with…

    7652b6e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

OS-macbuildThe build process and cross-buildtopic-JITtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions