Skip to content

Add an address-layer egress allowlist for hosted sandboxes - #73534

Merged
kaxil merged 1 commit into
apache:mainfrom
astronomer:sandbox-egress-cidr
Sep 22, 2026
Merged

kaxil merged 1 commit into
apache:mainfrom
astronomer:sandbox-egress-cidr

Conversation

@kaxil

@kaxil kaxil commented Sep 22, 2026

Copy link
Copy Markdown
Member

The Modal sandbox backend offered two egress modes. block_network=True is exact. A hostname allowlist is matched on the TLS handshake name, and it is refused unless the author opts in with egress_enforcement="sni", because it is weak in ways the spec does not show: TLS on 443 only, destination address ignored, name resolution open for every host. That left "deny everything or accept the caveats" as the only choice, when Modal has a third mode the backend never exposed. outbound_cidr_allowlist is enforced on the destination address for any port and protocol and denies by default outside the list.

This adds SandboxSpec.allow_egress_to_cidrs. On Modal it maps onto that allowlist and needs no opt-in, since there is no caveat for the author to accept. It is the mode for the case the hostname list serves worst and a platform team most often means by restricted egress: this sandbox may reach one internal service at a fixed address and nothing else. It cannot serve a package registry behind a CDN, whose addresses rotate faster than a sandbox lives, so the hostname list stays for that job. sbx refuses the field; its per-sandbox rule takes hostnames and there is nothing to map a range onto.

Measured on a live sandbox before writing the docs, because the open question on this design was whether name resolution survives an address-only list. With ["1.1.1.1/32"]: the listed address connected on 443 and on 53, an unlisted address timed out on both, and pypi.org still resolved through Modal's own resolver inside the sandbox. So hostnames resolve and then their addresses are unreachable. The tool description says exactly that, so the model does not read a successful lookup as a reachable host. It also means DNS is still a channel out under this mode, as under the hostname list; only block_network=True with no allowlist closes it, and the docs say so.

Entries are validated and normalised, since Modal passes the list through unchecked. A hostname, a URL or a host:port would be accepted by Modal and match nothing while reading as a restriction, so they are refused. A bare address is written as /32. A range with host bits set, such as 203.0.113.1/24, is refused rather than widened to 203.0.113.0/24, because that is not what the author wrote. 0.0.0.0/0 is refused as well: an allowlist of every address is an open network wearing a restriction's clothes, and block_network=False is the honest way to ask for one. IPv6 entries are refused too, with the reason: Modal's allowlist rejects them at create (Network access allowlist does not support IPv6 CIDRs) and the sandbox has no IPv6 route, so accepting one could only fail the task later and less legibly.

The two lists combine, and combining weakens the address one. Modal applies them additively, and I measured what that means: adding pypi.org to the hostname list beside ["1.1.1.1/32"] made a TCP connection to 8.8.8.8:443 succeed, because port 443 is then routed by handshake name for every address. A combined spec therefore has the address list's guarantee on every port except 443 and the hostname list's caveats there. The hostname half keeps its opt-in when combined, so adding an address list cannot launder the hostname list past it. The backend also logs a warning at create when both lists are set, naming which hostnames reopen port 443 and which ranges keep their guarantee elsewhere, so the weakening is symmetric and visible from both sides.

The destination has to be a public address. The natural use case reads as "our internal service", so I tested that too: connections to 10.20.0.1, 172.16.0.1, 192.168.1.1 and the cloud metadata address timed out from a Modal sandbox both under an open network and with those ranges on the allowlist. Private ranges are unreachable from a hosted sandbox whatever the spec says, so the field serves one service at a fixed public address, and the docs say so rather than promising an internal one.

Why a spec field rather than a backend parameter. The tool description reads the spec to tell the model what it can reach, and a network policy split across two objects is one the description would get wrong. A spec field also follows the contract's existing rule: a backend enforces it or refuses it, so a Dag moved to sbx fails loudly instead of running with an open network the author believed was closed.

The Modal system test gains a second task that checks the three measured facts on a real sandbox. No behaviour changes for specs that do not set the new field: the default, block_network=False, and the hostname list under both enforcement settings map exactly as before, and the existing tests for each still pass unchanged.

In the Airflow UI. One Dag, four tasks, the same probe from four sandboxes. Three report what they could and could not reach; the fourth is refused before a sandbox exists because it asks for the hostname allowlist without accepting its caveats.

Grid: three tasks succeed, the refused spec fails

The default spec denies everything, name resolution included:

deny_all_default: every destination blocked, names do not resolve

The address allowlist ["1.1.1.1/32"]: the listed address answers on 443 and 53, 8.8.8.8 times out, pypi.org and example.com resolve and are then unreachable:

address_allowlist: 1.1.1.1 reachable on both ports, everything else blocked, names still resolve

The hostname allowlist ["pypi.org"] under the opt-in: HTTPS to pypi.org works, example.com is reset at the handshake, plain HTTP to pypi.org fails, and raw TCP to any address on 443 connects because that port is routed by handshake name:

hostname_allowlist_sni: pypi.org over HTTPS only, example.com reset, plain HTTP blocked

The same hostname allowlist without the opt-in is refused at the first tool call, with the message naming the way out:

hostname_allowlist_refused: the backend refuses to provision and says why

Rendered docs. The Modal backend page gains the address allowlist as a third mode, with the measurements and the four caveats:

Modal backend page: the address allowlist section

The configuration page's network paragraph now names both lists and which backend enforces each:

Configuration page: the network paragraph

The backend comparison and the limitations list say where the field is refused and what it cannot serve:

What differs between the two backends: egress allowlists

Limitations: the two allowlists


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread providers/common/ai/tests/unit/common/ai/toolsets/test_sandbox.py Fixed
@kaxil
kaxil force-pushed the sandbox-egress-cidr branch from 161ec5f to 7490606 Compare September 22, 2026 11:01
The Modal backend offered two egress modes: deny everything, which is exact,
or a hostname allowlist matched on the TLS handshake name, which is weak in
ways the author cannot see and so needs an explicit opt-in. Modal has a third
mode the backend did not expose: an address allowlist enforced on the
destination for any port and protocol, deny-by-default outside the list.

SandboxSpec gains allow_egress_to_cidrs. On Modal it maps onto
outbound_cidr_allowlist and needs no opt-in, because there is no caveat to
accept; entries are validated and normalised to canonical CIDR form, since
Modal passes the list through unchecked and a hostname or a range with host
bits set would silently mean something other than what was written. It can be
combined with the hostname list, which Modal applies additively. sbx refuses
the field, having no per-sandbox address rule.

Measured live: a listed address connects on 443 and 53, an unlisted one is
dropped so the client times out, and hostnames still resolve through Modal's
own resolver. The tool description tells the model both facts, so a successful
lookup is not read as a reachable host. The system test gains a task that
checks all three on a real sandbox.
@kaxil
kaxil force-pushed the sandbox-egress-cidr branch from 7490606 to 83c68ee Compare September 22, 2026 11:09
@kaxil
kaxil marked this pull request as ready for review September 22, 2026 12:33
@kaxil
kaxil merged commit 55b8e9a into apache:main Sep 22, 2026
83 checks passed
@kaxil
kaxil deleted the sandbox-egress-cidr branch September 22, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants