Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -251,14 +251,14 @@ coffeeshop-cross-surface-equivalence-report:
uv run -- python -m benchbox.core.equivalence.cross_surface --benchmark coffeeshop

# Enforced gate: clickbench SQL<->DataFrame equivalence on a bounded DuckDB cell.
# In GATES (datavault remains staged) and run in the blocking correctness-gate (pr.yml);
# In GATES and run in the blocking correctness-gate (pr.yml);
# exits non-zero on any unclassified divergence. Q18's order-less LIMIT is the one
# classified exception (see _project/analysis/clickbench-cross-surface-divergences.md).
clickbench-cross-surface-equivalence-report:
uv run -- python -m benchbox.core.equivalence.cross_surface --benchmark clickbench

# Enforced gate: joinorder_synthetic SQL<->DataFrame equivalence on a bounded DuckDB
# cell. In GATES (datavault remains staged) and run in the blocking correctness-gate
# cell. In GATES and run in the blocking correctness-gate
# (pr.yml); exits non-zero on any unclassified divergence (see
# _project/analysis/joinorder-synthetic-cross-surface-divergences.md).
joinorder-synthetic-cross-surface-equivalence-report:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Cross-surface scope boundary and release-branch posture

Date: 2026-09-27

Status: Accepted.

## Scope boundary

Cross-surface gates (`benchbox/core/equivalence/cross_surface.py`, `GATES`)
cover benchmarks that ship both a SQL surface and a static `QueryRegistry`
DataFrame surface. That is transcription and regression verification against
Comment thread
joeharris76 marked this conversation as resolved.
DuckDB SQL references at a bounded equivalence scale: the two surfaces are
authored from the same understanding by the same person, so the gate catches
transcription drift, not shared conceptual errors, and its signal holds only
at the gated scale (see the module docstring and
`_project/analysis/cross-surface-oracle-independence.md` for per-benchmark
provenance).

Out of scope for the cross-surface gate, by construction:

- Operation-pipeline benchmarks with no DataFrame query surface route to the
w2 fallback oracle (differential second-engine check or curated
expected-results subset), not to a cross-surface builder:
`write_primitives`, `metadata_primitives`, `transaction_primitives`,
`tpcdi` (see `_project/analysis/cross-surface-applicability.md`).
- Dual-surface benchmarks that have a SQL surface plus a DataFrame query
registry but are not currently gated, per
`_project/analysis/cross-surface-applicability.md`: `joinorder` and
`nyctaxi` are not cheaply gateable (bounded-scale rejection, canonical
manifest fetch, or downloader-backed network fetch); `tpch_skew` and
`tsbs_devops` have unverified SQL-to-DataFrame ID mappings (zero verbatim
ID overlap, mapping must be confirmed independently, never guessed);
`tpcds_obt` has an abandoned correspondence (OBT-native Q1..Q17 versus
TPC-DS numbered SQL IDs, ruled out without renumbering one side). These
stay outside `GATES` until their named precondition is met; the
`joinorder_synthetic` CI-enforced gate covers scaled smoke-test data for
the JoinOrder family in the meantime.
- `ai_primitives` and `vector_search` stay `supports_dataframe: false` in
`benchmark_registry.yaml` and are likewise single-surface benchmarks
needing a fallback oracle, not cross-surface members.

Building the w2 fallback oracle itself is explicitly out of scope here; this
record only routes the benchmarks to it so they are not silently unguarded.

## Release-branch posture

`test.yml` runs only the bounded `test-correctness-gate` on release-bound
pull requests; the per-benchmark cross-surface suite runs in `pr.yml` on
`develop`. Release PRs therefore rely on develop-time squash-merge
enforcement: every change entering `develop` passes the blocking
correctness-gate suite (including all `GATES` cross-surface reports) before
it can ride a release. Adding the cross-surface suite to `test.yml` is a
separate, explicitly approved CI change if it is ever wanted; until then,
release coverage is by inheritance, not by omission.
2 changes: 1 addition & 1 deletion benchbox/core/equivalence/builders/datavault.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Data Vault cross-surface gate builder (staged)."""
"""Data Vault cross-surface gate builder."""

from __future__ import annotations

Expand Down
Loading