[smaheshwar-pltr]

Rationale for this change

Iceberg Java sends the DeleteObjects body to the S3 signing service, as the REST spec says to. PyIceberg doesn't. The body is the only place the keys being deleted appear, so REST catalogs like Lakekeeper reject DeleteObjects sign requests without it.

Since FsspecFileIO.delete sends DeleteObjects even for a single file, PyIceberg can't delete files through remote signing with these catalogs today.

Are these changes tested?

Yes. A new s3 test writes, reads, and deletes a file through S3V4RestSigner against the S3 container that the integration-test-s3 CI job runs. A requests_mock signing endpoint SigV4-signs each request with the container's credentials. The test checks that only the DeleteObjects sign request carries a body, and it fails without the fix.

Are there any user-facing changes?

Sign requests for DeleteObjects now include the request body.

AI Disclosure: This PR was authored by Claude Code (Opus 5.5) but directed and overseen by me.

[Fokko]

Thanks for raising this @smaheshwar-pltr. I know pre-signed URLs conceptually, but not the details. Maybe @danielcweeks has a free cycle to take a look at this. Can I ask if you have tested this against a S3 instance? One issue we have with the pre-signed url code is that it doesn't run any integration tests again MinIO.

[smaheshwar-pltr]

Mirroring Iceberg Java, which sends the DeleteObjects body here. See the PR description for motivation.

Java identifies DeleteObjects from the URL (POST with a delete query parameter). botocore already passes operation_name to every before-sign handler, so the signer now accepts it and checks it directly, with no URL parsing.

[smaheshwar-pltr]

Hey @Fokko, good to see you again! Thanks for taking a look, I've now opened this PR up for review so things should be clearer.

In response to #4039 (comment), we're dealing with remote signing here not pre-signed URLs, and I've since added an integration test against RustFS where the signing endpoint is a mock that lets us verify the the body is as expected. To me, this seems sufficient.

I've also updated the PR description to better describe the problem we're seeing currently 🙏

[kevinjqliu]

LGTM

I verified this locally against Lakekeeper with remote signing, comparing the PR's base commit with the PR head (2a008e3). The two checkouts differ only in the fsspec.py change.

Version	FileIO.delete
PR base	❌ SignError: Failed to sign request 400 on POST http://silo:9000/examples?delete. Lakekeeper logs DeleteWithoutBody: "Delete requests require a body"
This PR	✅ Delete succeeds and the file no longer exists

The client had no S3 credentials. Lakekeeper returned s3.signer=S3V4RestSigner, so every request, including the append, was remotely signed. The repro calls tbl.io.delete directly, which is the FsspecFileIO path this PR changes.

I also ran test_s3v4_rest_signer_write_read_delete with make test-s3: it passes on this PR and fails without the fix (assert 0 == 1).

Repro

1. Start Lakekeeper's minimal example (Lakekeeper, Postgres, Silo/MinIO). Only the catalog services are needed:

git clone --depth 1 https://github.com/lakekeeper/lakekeeper
cd lakekeeper/examples/minimal
docker compose up -d lakekeeper bootstrap initialwarehouse

2. Put repro.py (below) in a working directory, along with the pyiceberg/ source trees of the base commit and this PR:

mkdir -p repro/base repro/pr
git -C <iceberg-python> archive 2a008e3~1 pyiceberg | tar -x -C repro/base
git -C <iceberg-python> archive 2a008e3 pyiceberg | tar -x -C repro/pr

3. Run PyIceberg inside the compose network, since the warehouse's S3 endpoint is http://silo:9000. PYTHONPATH makes each run use its source tree instead of the installed pyiceberg:

docker run --rm --network minimal_iceberg_net -v "$PWD/repro":/w -w /w python:3.12-slim bash -c \
  "pip install -q 'pyiceberg[s3fs]' pyarrow && for v in base pr; do PYTHONPATH=/w/\$v python repro.py \$v; done"

4. repro.py:

import sys
import uuid

import pyarrow as pa

import pyiceberg
from pyiceberg.catalog import load_catalog

variant = sys.argv[1]
print(f"[{variant}] pyiceberg loaded from {pyiceberg.__file__}")

cat = load_catalog(
    "lk",
    **{
        "type": "rest",
        "uri": "http://lakekeeper:8181/catalog",
        "warehouse": "demo",
        "py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO",
        "header.X-Iceberg-Access-Delegation": "remote-signing",
    },
)
cat.create_namespace_if_not_exists("repro")
tbl = cat.create_table(f"repro.{variant}_{uuid.uuid4().hex[:8]}", pa.schema([("a", pa.int64())]))
props = tbl.io.properties
print(f"[{variant}] s3.signer={props.get('s3.signer')} has-access-key={'s3.access-key-id' in props}")

tbl.append(pa.table({"a": [1]}))
path = tbl.inspect.files().column("file_path")[0].as_py()
try:
    tbl.io.delete(path)
    print(f"[{variant}] DELETE OK, exists after delete: {tbl.io.new_input(path).exists()}")
except Exception as e:
    print(f"[{variant}] DELETE FAILED: {type(e).__name__}: {str(e)[:300]}")

Expected output:

[base] pyiceberg loaded from /w/base/pyiceberg/__init__.py
[base] s3.signer=S3V4RestSigner has-access-key=False
[base] DELETE FAILED: SignError: Failed to sign request 400: {'method': 'POST', 'region': 'local-01', 'uri': 'http://silo:9000/examples?delete', ...
[pr] pyiceberg loaded from /w/pr/pyiceberg/__init__.py
[pr] s3.signer=S3V4RestSigner has-access-key=False
[pr] DELETE OK, exists after delete: False

5. Tear down with docker compose down -v.

[kevinjqliu]

operation_name is passed to every before-sign handler by aiobotocore, which s3fs uses (See 2.4.2, 3.9.2).

[kevinjqliu]

Thanks @smaheshwar-pltr!

cc @danielcweeks feel free to take another look, we can fix forward if needed 😄

[kevinjqliu]

BTW I had my agent check against the source code of other catalogs.

Only Lakekeeper and Nessie support S3 remote signing; Polaris and Gravitino don't support it at all.
And only Lakekeeper reads the body, so the PR fixes deletes for Lakekeeper and makes no difference to Nessie, which ignores the field.

Catalog	Remote signing?	Effect of this PR
Lakekeeper	Yes. It parses the DeleteObjects XML and checks each key is inside the table location; with no body it rejects the request (DeleteWithoutBody).	Required, as confirmed by our local run.
Nessie	Yes. IcebergS3SignParams accepts an optional body and passes it to its signer if present. Authorization only checks the request URI against the table's locations. It never parses the delete keys.	No change. Nessie accepts the body as an optional field and ignores it.
Polaris	No. The S3 request-signing PRs (apache/polaris#2280, apache/polaris#3995) were closed without merging. A merged change (apache/polaris#5473) only clarified the error message for "unsupported remote signing". A feature request (apache/polaris#32) is still open.	None today.
Gravitino	No. Its Iceberg REST docs say "remote-signing: Gravitino doesn't support this mode yet". It only does credential vending.	None.