Skip to content

fix(azdls): route workload identity through azblob so signing succeeds - #10

Open
prakash5293 wants to merge 1 commit into
feat/replace-data-files-v3from
fix/azdls-workload-identity-tessellate
Open

prakash5293 wants to merge 1 commit into
feat/replace-data-files-v3from
fix/azdls-workload-identity-tessellate

Conversation

@prakash5293

Copy link
Copy Markdown
Collaborator

Fixes Azure Workload Identity for tessellate. Verified in production — see
below.

The symptom, and why it hid

Every metadata operation fails on Azure:

called: reqsign::Sign, service: azdls
=> signing http request, source: failed to load signing credential

The job still exits 0. That is what makes this worth fixing rather than
tolerating: v2 phase 2 (rebalance), phase 6 (hot-root) and the metadata census
fail on every tick while the CronJob reports success. Compaction settings that
look enabled are not running, and nothing in the exit code says so.

The existing wrap fires, and doesn't work

This branch already carries a bearer-token HTTP wrap for exactly this case, and
it does fire — enabling WI bearer-token http-client wrap appears once per
operator. The operation fails anyway:

26 wraps enabled, 26 failures — same run

opendal invokes reqsign's signer before the request reaches the HTTP client
the wrap replaced. The wrap patches a request on its way out; the failure
happens before there is a request to patch.

Cause

The existing comment already identifies it. opendal's azdls service builds
reqsign's context from a StaticEnv populated solely out of AzdlsConfig,
forwarding seven keys: account_name, account_key, sas_token, client_id,
client_secret, tenant_id, authority_host.

AZURE_FEDERATED_TOKEN_FILE is not among them and AzdlsConfig has no field
able to carry it
, so WorkloadIdentityCredentialProvider is structurally
blind. No adls.* property closes it.

Fix

opendal's azblob service builds the same context with OsEnv — the real process
environment — so the identical provider resolves the federated token unaided. An
ADLS Gen2 account serves both protocols, so routing this case through azblob
authenticates correctly against the same storage.

The endpoint is rewritten from the DFS host to the Blob host on that route: each
host speaks only its own API, and Blob requests at acct.dfs.core.windows.net
are rejected with 400 MissingRequiredHeader — authentication succeeds and the
write fails anyway. Covered by test_blob_endpoint_for.

Guarded on no static credential being supplied, so account_key / sas_token /
client_secret deployments keep their existing path. Non-Azure and
non-workload-identity deployments are untouched.

Verified against opendal 0.57.0 core/services/{azdls,azblob}/src/backend.rs
and reqsign-azure-storage 3.0.1
provide_credential/{default,workload_identity}.rs.

The wrap is left in place but is now unreachable for workload identity. It
can be removed once this route is confirmed — deliberately left for the author
to decide rather than deleted here.

Verified in production

Built as pr-264-4ddfe59 and deployed to qatest-qa-azure-test:

failed to load signing credential     26 → 0
v2 phase 2 (rebalance) failed          3 → 0
v2 phase 6 (hot-root) failed           3 → 0
metadata census failures               6 → 0
exit code                              0 (unchanged)

And the phases now do real work rather than merely not erroring:

rebalance phase A: rewrites_done=5 / rewrites_done=17
v2 phase 2 rebalance committed cleanly
v2 phase 7c: cold-paths sidecar rebuilt  leaves_scanned=279  paths_indexed=373
expired 2 snapshots

Then a full end-to-end pass on real data:

v2 phase 6 hot-root: partition compacted
  table=query_engine_history  removed=20  added=1  attempts=1  stuck=False
rowgroup_concat: total_rows=574  total_row_groups=20  chosen=Repack

20 fragmented files merged into 1; 148 distinct query IDs read back intact
afterwards.

Note for the reviewer

Same bug is fixed on the e6-iceberg-rust fork in two other PRs — one per
lineage, because opendal 0.57 and 0.58 differ in what hooks they expose. This is
Azure-only; on AWS the credential chain works and none of it is reachable, which
is why it was never caught upstream.

Tessellate cannot authenticate to ADLS under Azure Workload Identity.
Every metadata operation fails:

  called: reqsign::Sign, service: azdls
  => signing http request, source: failed to load signing credential

The bearer-token wrap added for this does fire -- "enabling WI
bearer-token http-client wrap" appears once per operator -- but the
operation fails anyway, because opendal invokes reqsign's signer BEFORE
the request reaches the HTTP client the wrap replaced. Observed on AKS:
26 wraps enabled and 26 failures in the same run. The wrap patches a
request on its way out; the failure happens before there is a request to
patch.

The cause is the one the existing comment already identifies:
opendal's azdls service builds reqsign's context from a StaticEnv
populated solely out of AzdlsConfig, which forwards seven keys --
account_name, account_key, sas_token, client_id, client_secret,
tenant_id, authority_host. AZURE_FEDERATED_TOKEN_FILE is not among them
and AzdlsConfig has no field able to carry it, so
WorkloadIdentityCredentialProvider is structurally blind. No adls.*
property can close the gap.

opendal's azblob service builds the same context with OsEnv -- the real
process environment -- so the identical provider resolves the federated
token unaided. An ADLS Gen2 account serves both protocols, so routing
this case through azblob authenticates correctly against the same
storage, using the library's own working path instead of signing by
hand.

The endpoint is rewritten from the DFS host to the Blob host when taking
that route. Each host speaks only its own API: Blob requests sent at
acct.dfs.core.windows.net are rejected with 400 MissingRequiredHeader,
so authentication succeeds and the write fails anyway. Only the first
`.dfs.` is replaced, so an account whose name contains "dfs" survives;
already-Blob and unrecognised endpoints pass through untouched, and the
rewrite is suffix-agnostic so sovereign clouds work. Covered by
test_blob_endpoint_for.

Guarded on no static credential being supplied: account_key, sas_token
and client_secret each select an explicit auth mode azdls already
handles, and keep their existing path. Non-Azure and non-workload-
identity deployments are unaffected.

The wrap is now unreachable for workload identity and can be removed
once this route is confirmed in production; left in place for the
reviewer to decide.

Verified against opendal 0.57.0 core/services/{azdls,azblob}/src/backend.rs
and reqsign-azure-storage 3.0.1
provide_credential/{default,workload_identity}.rs.

The same gap was fixed for laminar on the e6-iceberg-rust v0.10.1
lineage and verified in production: writes recovered, the wedge cleared,
and rows landed with current timestamps.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant