SAG (Setup-Agent)

SAG uses an LLM agent to configure a repository, provision its toolchain, and run builds and tests in Docker. It supports Maven, Gradle, and Python workflows. Every run leaves a record of what happened; the terminal, the written report, and the local Web Workbench all read that one record, so they cannot describe the same run differently.

You need Python 3.10+, uv, a running Docker daemon, and a configured model that supports native tool calling.

git clone https://github.com/Codegass/Setup-Agent.git
cd Setup-Agent
uv sync
cp .env.example .env

Edit .env: set SAG_ACTION_MODEL, SAG_ACTION_PROVIDER, and the matching provider API key. See .env.example for endpoint and other configuration options.

Start with a small project:

uv run sag project https://github.com/apache/commons-cli.git --record

This creates the sag-commons-cli container and clones the repository into /workspace/commons-cli. Use --ref <commit-or-tag> to choose a revision; use a full commit SHA for reproducible comparisons. --name changes the container name, not the repository directory.

While the run goes, the terminal prints one line per turn — what was asked, what came back, and how long it took — grouped by phase:

▸ provision
  #1   project   clone apache/commons-c…  e171117 → /workspace/commons-cli  4.8s
  #2   advisor   consult                  advice delivered                  4.9s
  #3   project   provision openjdk 8      java 1.8.0_502                   40.0s
  #4   project   provision maven 3.9.9    ok                                6.9s
  #5   phase     done success             workspace_present · workspace /…  0.4s
✓ provision advanced

When the run ends it prints the result card (seven rows, described below) and writes the report to the session directory under logs/. Read either again later with sag result and sag trajectory.

Open the Workbench to inspect workspaces, turns, evidence, and reports:

uv run sag ui --port 8765

Visit http://127.0.0.1:8765. The frontend is bundled; Node.js is needed only for frontend development. Keep Docker running for live workspace inspection and terminals. uv run sag ui --demo --port 8765 starts a preview with demo data.

Settings are read from environment variables and the repository's .env file.

Set the advisor window for the deployment you actually use. A blank override uses the local model catalog, with a disclosed fallback for unknown models. SAG_ADVISOR_SUMMARY_CONTEXT_WINDOW can separately bound the summarizer's window.

The actor may call advisor(question, context) with a short uncertainty and proposed next step; both arguments are optional. In on-demand mode the advisor receives a consultation-local catalog with source hashes. It can search literal text, read line ranges and continue long lines. These tools only inspect frozen recorded bytes. Missing live source or environment information must be obtained by the actor. Advice cannot execute commands, change requirements, or certify success; phases and the physical verifier remain authoritative.

A project setup follows five engine-owned phases:

provision → analyze → build → test → report

The model chooses tool calls within each phase. The engine handles dispatch, persisted evidence, phase transitions, and context rebuilding. A phase's "done" statement is checked against execution evidence before the run advances; unresolved work stays visible in the final result. sag run --task provides a separate, lighter flow for follow-up work in an existing container.

The execution path preserves information the models need to recover:

• Runtime evidence: JVM build receipts record the command, exit status, report paths and hashes, and observed toolchain. Dispatch observations survive retries and background settlement; a later shell environment does not certify an earlier run's version.
• Complete output access: large output is persisted with a retrieval reference and a concrete file path. The agent can inspect it through search, rg, or sed instead of relying only on the visible excerpt. Storage failures are disclosed.
• Advisor context: task requirements and current evidence, including bounded failure diagnostics, receive priority. Background material can be extracted or summarized; oversized protected material is reviewed in ordered slices. Original sources and compression records remain available.
• Report recovery: a missing report can be submitted through the repair flow after the evidence is closed, without reopening build execution or changing the recorded result.

Project files live in the container; the host retains orchestration records, session logs, and evidence publications. No host project directory is mounted by default.

Everything a run says about itself is derived from one append-only ledger, control_events.jsonl, which the engine writes on the host and mirrors into the container. It records every tool call the model asked for, every result, every phase decision and grading, and a record of each turn's own timing and token spend. Console logs are never a source: they show what the loop was doing, not what it recorded.

A --record session directory (logs/session_<timestamp>_<id>/) holds:

Two read-only derivations sit on top of the ledger and every surface uses them:

• The trajectory (sag trajectory, src/sag/trajectory/) folds the ledger into turns: for each turn, the call, its result, the gate's grading, its recorded span, and its token bill — the model's own response and, on turns that consulted the advisor, the advisor's spend, kept apart.
• The result card (sag result, src/sag/result_card/) states the run as seven rows in a fixed order. The terminal block, the written report, and the Workbench's result band are three renderings of that one card.

For implementation details, start with:

The result card has seven rows, always in this order: CI command, Official CI, Not confirmed by SAG, Run, Summary, Coverage, Report. The first four are measurements with a unit, a denominator and a source in the record; the Summary word is derived from them and labelled so. A row the run did not measure says so in words rather than showing a zero.

The same measurements are sealed in verdict.json (ci_command, official_ci, sag_checks, run_control, beside the word) and exposed as dimensions on the card API for the Workbench. A local success alone does not establish CI parity; the Official CI row carries the comparison, and only for the cell the run was handed.

Test failures and errors are reported separately. The pass rate uses non-skipped outcomes: passed / (passed + failed + errors). A value below 100% is never rounded into an unqualified 100%. outcomes accounted describes evidence accounting, not the fraction of tests that passed.

Static source declarations, class-file counts, and modules discovered on disk are diagnostics. They do not define the official build scope. unavailable means the evidence is missing or cannot support the comparison; it is not zero. A lower bound such as ≥N is not a complete total.

Tokens are two numbers, never one. The executing model's spend and the advisor's spend are recorded and shown apart — on the card (sag result --json: tokens_in/out and advisor_tokens_in/out), in the report's Tokens table, on the Workbench's Overview and per-turn strips. Where a total is shown it is labelled as the sum of the two rows above it.

When a run ends, the host renders setup-report-<timestamp>.md from the record and writes it into the session directory; it is also copied back into the container, best-effort, so /workspace/setup-report-*.md holds the same document. Its sections, in order:

1. Header — repository and commit, run id, version, and when the run ended.
2. Result — the seven-row card, CI command first and the derived Summary word last.
3. What was set up — the toolchain the run installed, with versions and paths, and any candidate it refused with the reason.
4. The run — one row per turn, grouped by phase: what was asked, what came back, how long it took.
5. Tokens — the model's and the advisor's calls and spend, apart.
6. Evidence — what the run cited, counted by kind, and the commands that open it.
7. Evidence accounting — how every test observation was accounted for.

A run with blockers lists them directly under the Result table. Every sentence traces to a field of the record; nothing the run did not measure is written.

CI comparison is integrated into live sag project runs. Supply the inputs before execution; finalization does not fetch CI or choose a different job after seeing the result.

• An acceptance task freezes the repository, full commit SHA, and ordered commands independently of the agent's plan. Steps can specify the launcher Java major and an exact Maven version. A task can be used without CI evidence.
• A CI target records the official job or matrix cell, commit, command, build scope, test outcomes, and source references. It supplies the comparison target, not proof that SAG executed anything.

After preparing task.json and target.json for the same revision:

uv run sag project https://github.com/apache/commons-cli.git \
  --acceptance-task-file task.json \
  --ci-target-file target.json \
  --record

The task selects its frozen commit automatically. If --ref is also supplied, it must equal that full SHA. See the AcceptanceTask and TargetRecord definitions for the input schemas. For a single Maven or Gradle command, --acceptance-command is also available and requires --ci-target-file.

Official evidence can be collected with d3_select_anchor.py and d3_harvest_target.py. small_ci_bench.py provides the download, freeze, prepare, and run workflow for CI-anchored experiments:

PYTHONPATH=. uv run python scripts/small_ci_bench.py --help

Missing CI scope receives no scope score. A missing target, unmatched matrix cell, revision mismatch, or incomplete evidence is shown with a reason; it must not be counted as CI attainment. Missing scope never supplies a fallback 1/1 score. Lifecycle-command parity is disclosed separately from scope attainment.

For model or strategy ablations, keep the project commits, task definitions, CI cells, resource limits, and run budgets fixed. Compare task completion and concrete build/test counts before time and cost. Equal test totals alone do not prove identical test coverage, and changing the evaluator does not constitute a new successful run.

The optional requirements v2 analysis separates compilation, packaging, installation, tests, documentation and quality checks within a frozen task. Add --requirements-file requirements.json together with an explicit full --ref, acceptance task and CI target. Reviewed definitions and recorded worktree/runtime evidence are required; draft definitions cannot start a formal campaign. SAG and the portable recorder share the offline scorer. Its sidecar does not replace the existing sealed verdict or reinterpret historical scores. See the recording and evaluation protocol for commands, intervention records, token-accounting limits and current readiness.

Primary comparisons must use agent-owned execution evidence. An external build after the agent exits measures environment readiness and cannot fill missing agent work. The campaign runner records the evidence location in primary-evidence.json. Native tools are retained: baseline hooks and a passive Maven process observer collect evidence without issuing build/test commands. The observer binds a pinned launch to the one open shell that names the step, also behind leading NAME=value prefixes, through a launcher named mvn-<version>, and for sequential pinned builds inside one shell call; Gradle and native-image steps still cannot bind. With that capture the baseline arms (Claude Code and OpenCode) ran the same 52 projects as SAG; the reports are under output/benchmark-fifty-20261021/ (SAG) and output/baselines-fifty-20261021/ (the comparison, with every non-passing cell classified by owner). Definitions whose requirement review is not complete are admitted with --allow-review-required: their task completion is scored and their requirement groups stay unavailable; scripts/freeze_benchmark_definitions.py builds such definitions from a project's CI log and POM. Historical 60-run scores retain their original post-agent acceptance meaning. See the execution-boundary qualification.

The research dataset guide links the versioned Java project registry, its selection rules, and evidence archives. It distinguishes candidate collection, admission as an official CI reference, and readiness for an agent experiment.

# List workspaces and enter a container
uv run sag list
uv run sag shell sag-commons-cli

# The run as turns: a table to read, or the whole trajectory as JSON
uv run sag trajectory logs/session_X
uv run sag trajectory logs/session_X --format json --detail full | jq .
uv run sag trajectory logs/session_X --follow          # tail a running session

# The result card, from a container or a recorded session
uv run sag result sag-commons-cli
uv run sag result logs/session_X --json | jq .

# One turn end to end, or a phase, from a container or a recorded session
uv run sag inspect sag-commons-cli --phase build
uv run sag inspect sag-commons-cli --session logs/session_X --turn 12

# Continue work, or remove the container and its filesystem
uv run sag run sag-commons-cli --task "Investigate the failing tests"
uv run sag remove sag-commons-cli

--record exports the container's /workspace/.setup_agent/ documents and the report into the session directory under logs/, beside the ledger the host wrote during the run. On the console, the turn stream owns the screen and the log prints only warnings; uv run sag --verbose project ... opens the full debug log on the console as well.

The Workbench shows the same record per workspace: the result band, then tabs for the Overview (time and tokens, anything that needs attention), Turns (every turn with its call, result, grading and bill, under three strips — model tokens, advisor tokens, and duration per turn), Tests, Build, Official CI, Evidence, Logs, and the Report.

For a version mismatch, inspect the failing invocation's dispatch observations and receipt. A plain java -version in a later shell may use a different environment. For missing counts or CI scores, read the associated evidence reason before attributing the result to the model.

Run uv run sag --help or uv run sag <command> --help for the complete CLI options.

uv sync
uv run pytest

# Frontend tests and bundled production assets
npm ci --prefix webui
npm test --prefix webui
npm run build --prefix webui

# Everything at once: vitest, the TypeScript build, the bundle, and pytest
uv run python scripts/ship_gate.py

The bundled frontend under src/sag/web/static/ is committed; rebuild it in the same commit as the source that changed it. Archived runs under tests/fixtures/trajectory/ and tests/fixtures/report_document/ replay real ledgers end to end, so a change to any surface is checked against what a run actually recorded, not only against hand-written fixtures.

Bug reports and pull requests should include the revision, command, runtime observations, and relevant receipts or logs so failures can be reproduced.

MIT.

BibTeX:

@inproceedings{Wei2026SAG,
  author    = {Wei, Chenhao and Zhao, Gengwu and Ye, Billy and Xiao, Lu and Li, Xinyi},
  title     = {Setup AGent (SAG): A Dual-Model LLM Agent for Autonomous End-to-End Java Project Configuration},
  booktitle = {Proceedings of the 48th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER '26)},
  year      = {2026},
  address   = {Rio de Janeiro, Brazil},
  publisher = {ACM},
  isbn      = {979-8-4007-2425-1},
  month     = apr,
  doi       = {10.1145/3786582.3786818}
}