fix(azdls): route workload identity through azblob so signing succeeds - #10
Open
prakash5293 wants to merge 1 commit into
Open
prakash5293 wants to merge 1 commit into
prakash5293 wants to merge 1 commit into
Conversation
Tessellate cannot authenticate to ADLS under Azure Workload Identity.
Every metadata operation fails:
called: reqsign::Sign, service: azdls
=> signing http request, source: failed to load signing credential
The bearer-token wrap added for this does fire -- "enabling WI
bearer-token http-client wrap" appears once per operator -- but the
operation fails anyway, because opendal invokes reqsign's signer BEFORE
the request reaches the HTTP client the wrap replaced. Observed on AKS:
26 wraps enabled and 26 failures in the same run. The wrap patches a
request on its way out; the failure happens before there is a request to
patch.
The cause is the one the existing comment already identifies:
opendal's azdls service builds reqsign's context from a StaticEnv
populated solely out of AzdlsConfig, which forwards seven keys --
account_name, account_key, sas_token, client_id, client_secret,
tenant_id, authority_host. AZURE_FEDERATED_TOKEN_FILE is not among them
and AzdlsConfig has no field able to carry it, so
WorkloadIdentityCredentialProvider is structurally blind. No adls.*
property can close the gap.
opendal's azblob service builds the same context with OsEnv -- the real
process environment -- so the identical provider resolves the federated
token unaided. An ADLS Gen2 account serves both protocols, so routing
this case through azblob authenticates correctly against the same
storage, using the library's own working path instead of signing by
hand.
The endpoint is rewritten from the DFS host to the Blob host when taking
that route. Each host speaks only its own API: Blob requests sent at
acct.dfs.core.windows.net are rejected with 400 MissingRequiredHeader,
so authentication succeeds and the write fails anyway. Only the first
`.dfs.` is replaced, so an account whose name contains "dfs" survives;
already-Blob and unrecognised endpoints pass through untouched, and the
rewrite is suffix-agnostic so sovereign clouds work. Covered by
test_blob_endpoint_for.
Guarded on no static credential being supplied: account_key, sas_token
and client_secret each select an explicit auth mode azdls already
handles, and keep their existing path. Non-Azure and non-workload-
identity deployments are unaffected.
The wrap is now unreachable for workload identity and can be removed
once this route is confirmed in production; left in place for the
reviewer to decide.
Verified against opendal 0.57.0 core/services/{azdls,azblob}/src/backend.rs
and reqsign-azure-storage 3.0.1
provide_credential/{default,workload_identity}.rs.
The same gap was fixed for laminar on the e6-iceberg-rust v0.10.1
lineage and verified in production: writes recovered, the wedge cleared,
and rows landed with current timestamps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes Azure Workload Identity for tessellate. Verified in production — see
below.
The symptom, and why it hid
Every metadata operation fails on Azure:
The job still exits 0. That is what makes this worth fixing rather than
tolerating: v2 phase 2 (rebalance), phase 6 (hot-root) and the metadata census
fail on every tick while the CronJob reports success. Compaction settings that
look enabled are not running, and nothing in the exit code says so.
The existing wrap fires, and doesn't work
This branch already carries a bearer-token HTTP wrap for exactly this case, and
it does fire —
enabling WI bearer-token http-client wrapappears once peroperator. The operation fails anyway:
opendal invokes reqsign's signer before the request reaches the HTTP client
the wrap replaced. The wrap patches a request on its way out; the failure
happens before there is a request to patch.
Cause
The existing comment already identifies it. opendal's azdls service builds
reqsign's context from a
StaticEnvpopulated solely out ofAzdlsConfig,forwarding seven keys:
account_name,account_key,sas_token,client_id,client_secret,tenant_id,authority_host.AZURE_FEDERATED_TOKEN_FILEis not among them andAzdlsConfighas no fieldable to carry it, so
WorkloadIdentityCredentialProvideris structurallyblind. No
adls.*property closes it.Fix
opendal's azblob service builds the same context with
OsEnv— the real processenvironment — so the identical provider resolves the federated token unaided. An
ADLS Gen2 account serves both protocols, so routing this case through azblob
authenticates correctly against the same storage.
The endpoint is rewritten from the DFS host to the Blob host on that route: each
host speaks only its own API, and Blob requests at
acct.dfs.core.windows.netare rejected with
400 MissingRequiredHeader— authentication succeeds and thewrite fails anyway. Covered by
test_blob_endpoint_for.Guarded on no static credential being supplied, so
account_key/sas_token/client_secretdeployments keep their existing path. Non-Azure andnon-workload-identity deployments are untouched.
Verified against opendal 0.57.0
core/services/{azdls,azblob}/src/backend.rsand reqsign-azure-storage 3.0.1
provide_credential/{default,workload_identity}.rs.The wrap is left in place but is now unreachable for workload identity. It
can be removed once this route is confirmed — deliberately left for the author
to decide rather than deleted here.
Verified in production
Built as
pr-264-4ddfe59and deployed toqatest-qa-azure-test:And the phases now do real work rather than merely not erroring:
Then a full end-to-end pass on real data:
20 fragmented files merged into 1; 148 distinct query IDs read back intact
afterwards.
Note for the reviewer
Same bug is fixed on the
e6-iceberg-rustfork in two other PRs — one perlineage, because opendal 0.57 and 0.58 differ in what hooks they expose. This is
Azure-only; on AWS the credential chain works and none of it is reachable, which
is why it was never caught upstream.