Recover stalled dashboard runs with guarded force-cancel

[trask]

Recover queued and waiting dashboard Actions runs that block newer requests in the same concurrency group, including partially completed runs with old, unassigned queued jobs. Recently queued, assigned, or started jobs remain protected.

The watchdog records normal cancellation attempts in a separate site-wide Netlify Blobs store. Only an accepted normal cancel that remains ineffective for 30 minutes can escalate to force-cancel, after fresh checks of both runs and all jobs. GitHub's final state determines confirmation; deleted runs are logged as unconfirmed and 409 retries are capped without promoting a rejected normal cancel to force. Backlog rotation bounds API work without pairing unrelated PR groups or manual runs.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Waiting on the author · refreshed 2026-09-26 02:24 UTC

Move out of draft to request review.

Status above doesn't look right?

• Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
• Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

[Copilot]

Copilot review overview

🟡 Changes recommended

Deleted runs can permanently block processing, and repeated 409 responses bypass the force-attempt ceiling.

Review effort: Balanced
Findings: 1

Open (1)

• Delete receipts for deleted workflow runs · New

What changed in this PR

Adds persisted watchdog state and guarded force-cancellation for stalled dashboard workflow runs.

Changes:

• Tracks cancellation requests and confirms final GitHub state.
• Revalidates runs/jobs before force-cancelling with retry and workload limits.
• Expands queued-job and GitHub API test coverage and documentation.

File	Description
WEBHOOK_SETUP.md	Documents cancellation escalation and confirmation.
test_workflow_watchdog.mjs	Tests persistence, force-cancellation, and safety checks.
test_github_dispatch.mjs	Tests run lookup and force-cancel API calls.
workflow-watchdog.mjs	Implements persistent guarded cancellation escalation.
github-dispatch.mjs	Adds run lookup and force-cancel client methods.
dashboard-workflow-watchdog.mjs	Connects the watchdog to its Blobs store.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Conflict responses bypass the retry cap, and deleted tracked runs can repeatedly abort watchdog processing.

Review effort: Balanced
Findings: 1

Open (1)

• Delete receipts for deleted workflow runs

Previously missed (1)

In code that hasn't changed since last review

409 conflicts bypass force retry tracking and cap

.github/​scripts/​pull-request-dashboard/​netlify/​lib/​workflow-watchdog.mjs:187

A 409 bypasses both forceAttempts and forceRequestedAt, so a still-listed run is retried every watchdog invocation forever rather than stopping after MAX_FORCE_ATTEMPTS. Record the attempt (or re-read and retire a completed run) on this path so conflicts cannot evade the retry cap.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Stateful force-cancellation of live workflows warrants final human validation despite the comprehensive safeguards and tests.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Delete receipts for deleted workflow runs