gh-132983: Introduce _zstd bindings module

[emmatyping]

This is part 2 (but parallel to part 1) for implementing PEP 784. This is probably the meat of the implementation making up ~half of the LOC in the overall diff.

This change:

• Adds the pyzstd license, since this is the first code introduced based on pyzstd
• Adds the code for the _zstd module in Modules/_zstd. Tests will be included when the Python compression.zstd module is added. If people think the tests should be included, I can merge the compression.zstd changes into the PR, but I wanted to separate them to make the PRs a bit more manageable.
• Adds unix build changes (Windows requires changes in cpython-source-deps and will be added in a follow-up PR)

I added skip news as I'd like to write a holistic NEWS/What's New entry once the entire implementation has landed. If people think each PR should have NEWS I can write something up.

TODO before merge:

• try running tests on top of this branch
• format PEP 7 style
• run buildbots on this PR
• verify _zstd gets installed

• Issue: Implement PEP 784 - Adding Zstandard to the Python standard library #132983

---

📚 Documentation preview 📚: https://cpython-previews--133027.org.readthedocs.build/

[emmatyping]

Hm, I'll dig into the C global failure. Didn't show up when the tests ran on the branch :(

E: now fixed

[ZeroIntensity]

Nice! Some comments at a glance.

Please remove the module state macros. They make this really difficult to review, and don't really add much (is MS_MEMBER(whatever) really that much easier to type than state->whatever?)

[emmatyping]

Thank you for the review @ZeroIntensity! Some of the decisions are just how things were in pyzstd, so I expect they very well should change for merging the code in. I'll go through your review and read up on ob_lock.

[ZeroIntensity]

There's probably not too much to read about it other than somewhere in PEP-703. You'll just want to replace the locks with critical sections, which have some magic to prevent deadlocks.

[picnixz]

Few preliminary comments, although I'm not sure it can be modified under the given license.

[picnixz]

Some other comments I managed to find. I didn't review the decompressor but I think most of my comments that I said for the compressor would apply. The most important ones are essentially function signatures where I want you to use PyObject *self and do a manual cast to the expected type inside the function:

#define MyType_CAST(op)	((MyType *)(op))

static void
my_method(PyObject *op) {
    MyType *self = MyType_CAST(op);
    ...
}

The use of a macro (or a static inline function, up to you) is to allow future runtime type checks.

[emmatyping]

Okay I think the items I have left to fix are:

• Format code to PEP 7
• Remove const from a few places
• Make sure that function signatures match their expected signatures (and cast as needed).

I'm going to try running the buildbots on this branch and I'll address the above and any buildbot issues later today. Then I will run the test_zstd suite on top of this PR in a branch.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @emmatyping for commit 1b9f786 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133027%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[emmatyping]

Buildbot failures look unrelated, so that's good!

[emmatyping]

Ah actually It looks like some of the failures are due to old zstd versions.

[emmatyping]

I suppose for the rhel 8 failures I can add a check for an old zstd and skip building it?

[emmatyping]

Okay, I believe I've sorted out all of the review comments. Thank you for the reviews! If I've missed anything, please let me know.

I'm going to make a new branch on top of this one that includes the compression.zstd module and tests and put up a different PR to run the tests on this code with the buildbots.

[emmatyping]

I opened #133185 to test this code with the Python tests (which will be added in a follow up)