From df59d66fb1f979529f3b4c782a8a49892f9bcd59 Mon Sep 17 00:00:00 2001 From: Joe Harris Date: Sun, 27 Sep 2026 11:30:48 -0400 Subject: [PATCH 1/6] docs: close cross-surface residual wording, scope, and release posture Remove stale datavault staged claims from Makefile comments and the builder docstring. Record the scope boundary routing operation-pipeline benchmarks to the w2 fallback oracle and the release-branch posture relying on develop-time enforcement. --- Makefile | 4 +- ...ce-scope-and-release-posture-2026-09-27.md | 42 +++++++++++++++++++ .../core/equivalence/builders/datavault.py | 2 +- 3 files changed, 45 insertions(+), 3 deletions(-) create mode 100644 _project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md diff --git a/Makefile b/Makefile index f0abcbcd0e..6b1d97f2d7 100644 --- a/Makefile +++ b/Makefile @@ -251,14 +251,14 @@ coffeeshop-cross-surface-equivalence-report: uv run -- python -m benchbox.core.equivalence.cross_surface --benchmark coffeeshop # Enforced gate: clickbench SQL<->DataFrame equivalence on a bounded DuckDB cell. -# In GATES (datavault remains staged) and run in the blocking correctness-gate (pr.yml); +# In GATES and run in the blocking correctness-gate (pr.yml); # exits non-zero on any unclassified divergence. Q18's order-less LIMIT is the one # classified exception (see _project/analysis/clickbench-cross-surface-divergences.md). clickbench-cross-surface-equivalence-report: uv run -- python -m benchbox.core.equivalence.cross_surface --benchmark clickbench # Enforced gate: joinorder_synthetic SQL<->DataFrame equivalence on a bounded DuckDB -# cell. In GATES (datavault remains staged) and run in the blocking correctness-gate +# cell. In GATES and run in the blocking correctness-gate # (pr.yml); exits non-zero on any unclassified divergence (see # _project/analysis/joinorder-synthetic-cross-surface-divergences.md). joinorder-synthetic-cross-surface-equivalence-report: diff --git a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md new file mode 100644 index 0000000000..8e60c5cd97 --- /dev/null +++ b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md @@ -0,0 +1,42 @@ +# Cross-surface scope boundary and release-branch posture + +Date: 2026-09-27 + +Status: Accepted. + +## Scope boundary + +Cross-surface gates (`benchbox/core/equivalence/cross_surface.py`, `GATES`) +cover benchmarks that ship both a SQL surface and a static `QueryRegistry` +DataFrame surface. That is transcription and regression verification against +DuckDB SQL references at a bounded equivalence scale: the two surfaces are +authored from the same understanding by the same person, so the gate catches +transcription drift, not shared conceptual errors, and its signal holds only +at the gated scale (see the module docstring and +`_project/analysis/cross-surface-oracle-independence.md` for per-benchmark +provenance). + +Out of scope for the cross-surface gate, by construction: + +- Operation-pipeline benchmarks with no DataFrame query surface route to the + w2 fallback oracle (differential second-engine check or curated + expected-results subset), not to a cross-surface builder: + `write_primitives`, `metadata_primitives`, `transaction_primitives`, + `tpcdi` (see `_project/analysis/cross-surface-applicability.md`). +- `ai_primitives` and `vector_search` stay `supports_dataframe: false` in + `benchmark_registry.yaml` and are likewise single-surface benchmarks + needing a fallback oracle, not cross-surface members. + +Building the w2 fallback oracle itself is explicitly out of scope here; this +record only routes the benchmarks to it so they are not silently unguarded. + +## Release-branch posture + +`test.yml` runs only the bounded `test-correctness-gate` on release-bound +pull requests; the per-benchmark cross-surface suite runs in `pr.yml` on +`develop`. Release PRs therefore rely on develop-time squash-merge +enforcement: every change entering `develop` passes the blocking +correctness-gate suite (including all `GATES` cross-surface reports) before +it can ride a release. Adding the cross-surface suite to `test.yml` is a +separate, explicitly approved CI change if it is ever wanted; until then, +release coverage is by inheritance, not by omission. diff --git a/benchbox/core/equivalence/builders/datavault.py b/benchbox/core/equivalence/builders/datavault.py index 0f267efb24..472a8d8d23 100644 --- a/benchbox/core/equivalence/builders/datavault.py +++ b/benchbox/core/equivalence/builders/datavault.py @@ -1,4 +1,4 @@ -"""Data Vault cross-surface gate builder (staged).""" +"""Data Vault cross-surface gate builder.""" from __future__ import annotations From 5ee2180dae10c703ce284b15bd52177bfa98c151 Mon Sep 17 00:00:00 2001 From: Joe Harris Date: Sun, 27 Sep 2026 23:52:19 -0400 Subject: [PATCH 2/6] docs: enumerate dual-surface benchmarks excluded from cross-surface gates The scope boundary listed only single-surface benchmarks routed to the w2 fallback oracle, implying every SQL plus QueryRegistry benchmark is covered by GATES. Name the five that remain outside: joinorder and nyctaxi are not cheaply gateable, tpch_skew and tsbs_devops have unverified ID mappings, and tpcds_obt has an abandoned correspondence. --- ...s-surface-scope-and-release-posture-2026-09-27.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md index 8e60c5cd97..7e68d01b7c 100644 --- a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md +++ b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md @@ -23,6 +23,18 @@ Out of scope for the cross-surface gate, by construction: expected-results subset), not to a cross-surface builder: `write_primitives`, `metadata_primitives`, `transaction_primitives`, `tpcdi` (see `_project/analysis/cross-surface-applicability.md`). +- Dual-surface benchmarks that have a SQL surface plus a DataFrame query + registry but are not currently gated, per + `_project/analysis/cross-surface-applicability.md`: `joinorder` and + `nyctaxi` are not cheaply gateable (bounded-scale rejection, canonical + manifest fetch, or downloader-backed network fetch); `tpch_skew` and + `tsbs_devops` have unverified SQL-to-DataFrame ID mappings (zero verbatim + ID overlap, mapping must be confirmed independently, never guessed); + `tpcds_obt` has an abandoned correspondence (OBT-native Q1..Q17 versus + TPC-DS numbered SQL IDs, ruled out without renumbering one side). These + stay outside `GATES` until their named precondition is met; the + `joinorder_synthetic` CI-enforced gate covers scaled smoke-test data for + the JoinOrder family in the meantime. - `ai_primitives` and `vector_search` stay `supports_dataframe: false` in `benchmark_registry.yaml` and are likewise single-surface benchmarks needing a fallback oracle, not cross-surface members. From 7d9fb7666f6c879d0baa776be8251619c19e83ad Mon Sep 17 00:00:00 2001 From: Joe Harris Date: Thu, 1 Oct 2026 08:30:37 -0400 Subject: [PATCH 3/6] Correct the cross-surface scope record's routing and release-posture claims State that the fallback oracle for operation-pipeline benchmarks does not exist yet instead of implying the record routes to it, defer per-benchmark gate status to cross_surface.py and the oracle coverage map, name ci.yml, and record that a direct release-branch hotfix is not covered by develop-time enforcement. --- ...ce-scope-and-release-posture-2026-09-27.md | 53 +++++++++---------- 1 file changed, 25 insertions(+), 28 deletions(-) diff --git a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md index 7e68d01b7c..d2ff719b2c 100644 --- a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md +++ b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md @@ -16,39 +16,36 @@ at the gated scale (see the module docstring and `_project/analysis/cross-surface-oracle-independence.md` for per-benchmark provenance). -Out of scope for the cross-surface gate, by construction: - -- Operation-pipeline benchmarks with no DataFrame query surface route to the - w2 fallback oracle (differential second-engine check or curated - expected-results subset), not to a cross-surface builder: - `write_primitives`, `metadata_primitives`, `transaction_primitives`, - `tpcdi` (see `_project/analysis/cross-surface-applicability.md`). -- Dual-surface benchmarks that have a SQL surface plus a DataFrame query - registry but are not currently gated, per - `_project/analysis/cross-surface-applicability.md`: `joinorder` and - `nyctaxi` are not cheaply gateable (bounded-scale rejection, canonical - manifest fetch, or downloader-backed network fetch); `tpch_skew` and - `tsbs_devops` have unverified SQL-to-DataFrame ID mappings (zero verbatim - ID overlap, mapping must be confirmed independently, never guessed); - `tpcds_obt` has an abandoned correspondence (OBT-native Q1..Q17 versus - TPC-DS numbered SQL IDs, ruled out without renumbering one side). These - stay outside `GATES` until their named precondition is met; the - `joinorder_synthetic` CI-enforced gate covers scaled smoke-test data for - the JoinOrder family in the meantime. +Which dual-surface benchmarks are enforced (`GATES`) or staged +(`STAGED_GATES`) changes as burn-downs finish, so this record does not list +them; `cross_surface.py` and `_project/analysis/oracle-coverage-map.md` are the +authority. Out of scope for the cross-surface gate, by construction: + +- Operation-pipeline benchmarks with no DataFrame query surface need a + different oracle (a differential second-engine check or a curated + expected-results subset), not a cross-surface builder: `write_primitives`, + `metadata_primitives`, `transaction_primitives`, `tpcdi` (see + `_project/analysis/cross-surface-applicability.md`). That oracle does not + exist yet, so until it does these benchmarks have no cross-surface + protection; this record does not provide it. +- `joinorder` stays outside `GATES`: it accepts only the canonical IMDb data at + `scale_factor=1.0`, which is not a bounded routine-PR cell. The + `joinorder_synthetic` CI-enforced gate covers scaled smoke-test data for the + JoinOrder family in the meantime. +- `tpcds_obt` has an abandoned correspondence (OBT-native Q1..Q17 versus + TPC-DS numbered SQL IDs), ruled out without renumbering one side. - `ai_primitives` and `vector_search` stay `supports_dataframe: false` in - `benchmark_registry.yaml` and are likewise single-surface benchmarks - needing a fallback oracle, not cross-surface members. - -Building the w2 fallback oracle itself is explicitly out of scope here; this -record only routes the benchmarks to it so they are not silently unguarded. + `benchmark_registry.yaml` and are single-surface benchmarks that need the + same non-cross-surface oracle. ## Release-branch posture `test.yml` runs only the bounded `test-correctness-gate` on release-bound -pull requests; the per-benchmark cross-surface suite runs in `pr.yml` on +pull requests; the per-benchmark cross-surface suite runs in `ci.yml` on `develop`. Release PRs therefore rely on develop-time squash-merge enforcement: every change entering `develop` passes the blocking correctness-gate suite (including all `GATES` cross-surface reports) before -it can ride a release. Adding the cross-surface suite to `test.yml` is a -separate, explicitly approved CI change if it is ever wanted; until then, -release coverage is by inheritance, not by omission. +it can ride a release. A change committed directly to a release branch without +passing through `develop` (a hotfix) is not covered by that inheritance and +gets only `test-correctness-gate`. Adding the cross-surface suite to +`test.yml` is a separate, explicitly approved CI change if it is ever wanted. From a8cc63c3c7b706c2140b18f2dfd720b13289fe87 Mon Sep 17 00:00:00 2001 From: Joe Harris Date: Thu, 1 Oct 2026 08:32:11 -0400 Subject: [PATCH 4/6] State the release-branch CI coverage the workflows actually provide ci.yml runs on pull requests against any base and the release ruleset rejects direct pushes, so release-bound changes are not covered only by inheritance from develop. Also record that canonical-data-only JoinOrder divergences are outside every cross-surface gate. --- ...ce-scope-and-release-posture-2026-09-27.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md index d2ff719b2c..c3bdec7fca 100644 --- a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md +++ b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md @@ -31,7 +31,8 @@ authority. Out of scope for the cross-surface gate, by construction: - `joinorder` stays outside `GATES`: it accepts only the canonical IMDb data at `scale_factor=1.0`, which is not a bounded routine-PR cell. The `joinorder_synthetic` CI-enforced gate covers scaled smoke-test data for the - JoinOrder family in the meantime. + JoinOrder family in the meantime. A divergence that appears only on the + canonical IMDb distributions is not caught by any cross-surface gate. - `tpcds_obt` has an abandoned correspondence (OBT-native Q1..Q17 versus TPC-DS numbered SQL IDs), ruled out without renumbering one side. - `ai_primitives` and `vector_search` stay `supports_dataframe: false` in @@ -40,12 +41,11 @@ authority. Out of scope for the cross-surface gate, by construction: ## Release-branch posture -`test.yml` runs only the bounded `test-correctness-gate` on release-bound -pull requests; the per-benchmark cross-surface suite runs in `ci.yml` on -`develop`. Release PRs therefore rely on develop-time squash-merge -enforcement: every change entering `develop` passes the blocking -correctness-gate suite (including all `GATES` cross-surface reports) before -it can ride a release. A change committed directly to a release branch without -passing through `develop` (a hotfix) is not covered by that inheritance and -gets only `test-correctness-gate`. Adding the cross-surface suite to -`test.yml` is a separate, explicitly approved CI change if it is ever wanted. +`ci.yml` has no branch filter on `pull_request`, so a pull request opened +against `release` runs the same path-aware unit checks as one against +`develop`, including the blocking correctness-gate suite and its `GATES` +cross-surface reports when the touched paths select them. `test.yml` +additionally runs the bounded `test-correctness-gate` on release-bound pull +requests. The `release` ruleset rejects direct pushes, so every change reaches +it through a pull request. Adding the full cross-surface suite to `test.yml` is +a separate, explicitly approved CI change if it is ever wanted. From 0bff9afd50865cba16939312afc0cd523942ef94 Mon Sep 17 00:00:00 2001 From: Joe Harris Date: Thu, 1 Oct 2026 17:20:48 -0400 Subject: [PATCH 5/6] docs: separate develop-time cross-surface enforcement from release gating The release-branch posture said a release pull request runs the blocking cross-surface suite. The release ruleset requires only validate-base and release-required-result, and the latter aggregates test.yml, not ci.yml's core unit or its cross-surface reports, so nothing makes them merge-blocking on a release pull request. State that, name the bounded gate test.yml does run, and record that release curation removes the script ci.yml runs unconditionally on every pull request. --- ...ce-scope-and-release-posture-2026-09-27.md | 29 ++++++++++++++----- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md index c3bdec7fca..08c9b9eb2e 100644 --- a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md +++ b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md @@ -41,11 +41,26 @@ authority. Out of scope for the cross-surface gate, by construction: ## Release-branch posture +Cross-surface enforcement is a develop-time gate. It is not a release-time +gate. + `ci.yml` has no branch filter on `pull_request`, so a pull request opened -against `release` runs the same path-aware unit checks as one against -`develop`, including the blocking correctness-gate suite and its `GATES` -cross-surface reports when the touched paths select them. `test.yml` -additionally runs the bounded `test-correctness-gate` on release-bound pull -requests. The `release` ruleset rejects direct pushes, so every change reaches -it through a pull request. Adding the full cross-surface suite to `test.yml` is -a separate, explicitly approved CI change if it is ever wanted. +against `release` triggers the same workflow as one against `develop`, but the +`release` ruleset does not require its results. The required checks are +`validate-base` and `release-required-result` +(`docs/operations/repo-admin-settings.md`). `release-required-result` +aggregates `test.yml`, which runs the bounded `test-correctness-gate` +(`make test-correctness-gate`, a strict expected-results run of the local +platform benchmark matrix) and does not include `ci.yml`'s `core` unit or its +`GATES` cross-surface reports. Those reports can run on a release pull request, +but nothing makes them merge-blocking there. + +The `release` ruleset rejects direct pushes, so every change reaches it through +a pull request. Making cross-surface coverage a release requirement would be a +separate, explicitly approved change to `test.yml` or the ruleset. + +Known limitation: release curation (`make release-cut`) removes +`_project/scripts/todo_state_contract_check.py`, while `ci.yml` runs it on every +pull request without a condition. As written, that step cannot succeed on a +curated release tree, so `ci.yml` results on release pull requests are not a +usable signal until the two are reconciled. This record does not change either. From d95bd47d3af782bebfdd5dbfaa4cd2f599ac3f03 Mon Sep 17 00:00:00 2001 From: Joe Harris Date: Thu, 1 Oct 2026 22:30:30 -0400 Subject: [PATCH 6/6] docs: correct the correctness-gate scope and note a stale staging comment The release posture described test-correctness-gate as a run of the local platform benchmark matrix. It runs one case, TPC-H on DuckDB, over the gated query ids. Say so. Record that the comment above STAGED_GATES still lists tpcds_obt as a next gateable benchmark although the correspondence was abandoned, and qualify the single-author statement, since how separately the two surfaces were handwritten varies by benchmark. --- ...ace-scope-and-release-posture-2026-09-27.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md index 08c9b9eb2e..72cc2f92b1 100644 --- a/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md +++ b/_project/decisions/cross-surface-scope-and-release-posture-2026-09-27.md @@ -10,9 +10,10 @@ Cross-surface gates (`benchbox/core/equivalence/cross_surface.py`, `GATES`) cover benchmarks that ship both a SQL surface and a static `QueryRegistry` DataFrame surface. That is transcription and regression verification against DuckDB SQL references at a bounded equivalence scale: the two surfaces are -authored from the same understanding by the same person, so the gate catches -transcription drift, not shared conceptual errors, and its signal holds only -at the gated scale (see the module docstring and +authored from the same understanding by the same person (how separately they +were handwritten varies by benchmark, as each gate's `surface_independence` +records), so the gate catches transcription drift, not shared conceptual +errors, and its signal holds only at the gated scale (see the module docstring and `_project/analysis/cross-surface-oracle-independence.md` for per-benchmark provenance). @@ -34,7 +35,10 @@ authority. Out of scope for the cross-surface gate, by construction: JoinOrder family in the meantime. A divergence that appears only on the canonical IMDb distributions is not caught by any cross-surface gate. - `tpcds_obt` has an abandoned correspondence (OBT-native Q1..Q17 versus - TPC-DS numbered SQL IDs), ruled out without renumbering one side. + TPC-DS numbered SQL IDs), ruled out without renumbering one side. The + comment above `STAGED_GATES` in `cross_surface.py` still lists `tpcds_obt` + among the next gateable benchmarks; it predates this decision and is + stale. - `ai_primitives` and `vector_search` stay `supports_dataframe: false` in `benchmark_registry.yaml` and are single-surface benchmarks that need the same non-cross-surface oracle. @@ -50,9 +54,9 @@ against `release` triggers the same workflow as one against `develop`, but the `validate-base` and `release-required-result` (`docs/operations/repo-admin-settings.md`). `release-required-result` aggregates `test.yml`, which runs the bounded `test-correctness-gate` -(`make test-correctness-gate`, a strict expected-results run of the local -platform benchmark matrix) and does not include `ci.yml`'s `core` unit or its -`GATES` cross-surface reports. Those reports can run on a release pull request, +(`make test-correctness-gate`, a strict expected-results run of TPC-H on +DuckDB over the gated query ids in `CORRECTNESS_GATE_QUERY_IDS`) and does not +include `ci.yml`'s `core` unit or its `GATES` cross-surface reports. Those reports can run on a release pull request, but nothing makes them merge-blocking there. The `release` ruleset rejects direct pushes, so every change reaches it through