This centrally executed workflow collects one compact timing record for every
terminal GitHub Actions job in active public repositories under
open-telemetry. Records are written to the orphan
otelbot/github-actions-queue-data branch in this repository and can be
queried to find queue-time outliers.
Data writers always retain only the latest parentless Git snapshot of the data branch. All historical job measurements in the current files, collection checkpoints, and report inputs remain intact.
The snapshot protocol is unconditional and no longer needs
DATA_BRANCH_SNAPSHOTS. The obsolete repository variable can be deleted after
the workflow environment cleanup is merged. See the
shared storage requirements.
The hourly collection and Pages deployment schedules do not change.
Queue time is measured per job:
queue_seconds = job_started_at - job_created_at
Matrix jobs are separate records. Jobs that never receive a runner or have a negative queue interval are not stored because they have no usable runner queue time. This excludes skipped jobs, jobs cancelled before assignment, and check runs created only to publish results.
When only failed jobs are rerun, GitHub also returns cloned records for jobs
that did not execute again. The clones have a later created_at but retain the
original execution timestamps, producing a negative queue interval that the
collector discards.
The hourly queue-time dashboard is published at https://open-telemetry.github.io/shared-workflows/github-actions-queue/. It opens to the latest seven days of GitHub-hosted runner data and provides:
- Required GitHub-hosted or self-hosted runner selection
- Runner labels scoped to the selected host category
- Repository filtering
- Inclusive UTC start and end dates that default to the latest seven days
- Shareable URLs that retain the selected filters
- Hourly p50, p90, p95, and p99 queue times in UTC
- Exact p50, p90, p95, and p99 queue times across the selected range
The self-hosted category matches runner labels case-insensitively against:
self-hosted
cncf-*
oracle-*
*-s390x
These broad patterns cover the special self-hosted runners documented in the
OpenTelemetry community asset inventory.
All other assigned runners are classified as GitHub-hosted. In particular,
otel-windows-latest-8-cores is GitHub-hosted even though it uses an
organization-defined runner group.
The collector incrementally updates report-state.json.gz and daily report
partitions under report-data/ on the data branch. The Pages artifact includes
only the dashboard assets and those derived partitions, not the raw job files
or collector checkpoints.
Repository administrators must enable GitHub Pages once in the repository
settings and select GitHub Actions as the source. The deployment uses the
standard github-pages environment and needs no new GitHub App permission,
token, or secret.
Use the private organization-owned GitHub App named
OpenTelemetry Actions Telemetry. Its homepage is
https://github.com/open-telemetry/shared-workflows, and it has no webhook or
event subscriptions.
The App requires only:
- Actions: read-only
- Metadata: read-only, granted automatically
Install it for All repositories so one installation token can read Actions
data across the organization and new repositories are covered automatically.
It requests no webhook events and no write permissions. The collector lists
only public repositories and rejects an explicitly selected repository unless
GitHub reports private: false.
Store its credentials in the protected environment of shared-workflows:
- Variable
ACTIONS_TELEMETRY_CLIENT_ID - Secret
ACTIONS_TELEMETRY_PRIVATE_KEY
The workflow creates a short-lived installation token for API reads. Its
built-in GITHUB_TOKEN, scoped to shared-workflows, writes the data branch.
.github/workflows/github-actions-queue-collector.yml runs at minute 17 of
every hour. It discovers active public repositories dynamically, then processes
closed one-hour workflow-run windows. A repository checkpoint lets the next run
resume an incomplete window without re-emitting repositories already committed.
The collector prioritizes live data, then uses the remaining run budget to
backfill from 2026-09-17T02:00:00Z toward 2026-01-01T00:00:00Z. Backfill
processes newer hours first and continues automatically on each schedule.
backfill-state.json stores the oldest complete hour plus partial repository
progress, independently of the live state.json cursor.
Up to four repositories are processed concurrently. Workflow invocations stop after at most 20 total windows, 12,000 requests, or 40 minutes of collection. These limits leave room below the App's 15,000-request hourly quota and the workflow's 50-minute timeout. A stopped run commits its records and checkpoints before the next schedule resumes it.
Runs that have not reached a terminal state are saved in state.json. Later
collections revisit them and emit their jobs only after the parent run is
completed and every runner-assigned job is terminal. Runnerless jobs are
excluded before this completion check, so they do not block measurable sibling
jobs. Completed runs containing only runnerless jobs leave the pending set
without emitting records.
A job reported as in_progress or queued is also terminal when GitHub
supplies both a completion timestamp and a terminal conclusion. Such assigned
jobs missing either piece of evidence still block the entire run. Records retain
GitHub's reported job status and conclusion.
Job listing uses filter=all, so actual executions from every
available attempt are retained when the run is finalized. Carry-forward clones
created by partial reruns are not stored.
Each live or backfill collection selects up to 25 saved pending runs before collecting windows. Five slots are reserved for unresolved API failures and 20 for other unfinished runs; unused slots are shared. Both groups use the oldest last-attempt time, falling back to creation time for unattempted runs. Selected runs are processed in that same chronological order, within the request, runtime, and REST quota limits. Backfill retries run only after live collection catches up without pausing and leaves a window budget.
The collector tracks the installation's REST quota. If it exhausts the quota, it commits the partial checkpoint and resumes during the next scheduled run.
The completed runnerless and negative-queue cleanup is recorded in
migrations/remove-unmeasurable-jobs-v1.json on the data branch.
GitHub limits a filtered workflow-run search to 1,000 results. The collector splits busy one-hour windows into smaller ranges until each result can be paginated safely. Repository and workflow-run pages contain up to 100 results. Job pages contain up to 50 results to reduce GitHub jobs-API server errors on large pages. The collector checks that all pages yield the reported number of unique jobs before finalizing a run.
Each successful collection adds one immutable gzip-compressed JSON Lines file:
jobs/date=2026-09-15/collection-123456789-1.jsonl.gz
state.json and backfill-state.json are mutable collector checkpoints, not
reporting data. Each job line contains:
| Field | Meaning |
|---|---|
schema_version |
Job-record schema version. |
organization |
GitHub organization, currently open-telemetry. |
repository |
Repository name under open-telemetry. |
repository_id |
Stable GitHub repository ID when the runs API supplies it. |
workflow_name, workflow_id |
Workflow identity. |
run_id, run_attempt, run_created_at |
Workflow run identity and creation time. |
event, from_fork |
Trigger and whether the head repository was a fork. from_fork is null when GitHub no longer provides the head repository. |
head_branch, head_sha |
Source revision. |
job_id, job_name |
Individual job identity. Matrix values normally appear in job_name. |
job_status, job_conclusion |
GitHub's reported state and result. A terminal job can retain a stale in_progress or queued status. |
job_created_at, job_started_at, job_completed_at |
GitHub job timestamps. |
queue_seconds |
Start minus creation time. |
runner_assigned |
Always true for retained queue records. |
runner_labels, runner_name, runner_group_name |
Runner classification. |
html_url |
Direct link to the job. |
collected_at |
Time the collector finalized the record. |
The format deliberately retains self-hosted jobs, actual retries, fork runs, and large queue values. Reports should filter those dimensions rather than discarding raw records during collection.
Set GH_TOKEN, GITHUB_AUTH_TOKEN, or GITHUB_TOKEN, then use a temporary
state and output directory:
python3 .github/scripts/github-actions-queue/collect.py \
--repository shared-workflows \
--start 2026-09-15T00:00:00Z \
--state /tmp/actions-queue-state.json \
--output-dir /tmp/actions-queue-jobs \
--collection-id local-test \
--max-windows 1The command prints a JSON summary containing the request count plus starting and
ending limit, remaining, and reset values. The starting snapshot comes
from the first ordinary API response because GitHub's /rate_limit response
can lag the bucket reported by repository endpoints. The summary also reports
the lowest remaining fraction observed because GitHub can return fluctuating
quota headers across sequential public-repository requests.
The summary's exclusions contains runnerless_jobs, runnerless_runs, and
negative_queue_jobs counts across live and backfill collection, including
pending retries. runnerless_runs counts completed, nonempty runs containing
only runnerless jobs. Counts cover finalized runs in checkpointed repository
windows and resolved pending retries, not runs still waiting for assigned jobs
or discarded partial repository work.
The runs API filters by original creation time, not the time of a later rerun.
The collector preserves all attempts returned when it finalizes a discovered
run, but a rerun requested after that run leaves the pending set can fall
outside later creation-time windows. Closing this gap may require a bounded
lookback or workflow_job webhook ingestion.
Failed job lookups are retained in state.json with their failure count, last
error, and last attempt time. Backfill failures are retained in
backfill-state.json the same way. Later collections retry them while
continuing to discover other runs. The workflow emits a warning as long as any
failed lookup remains unresolved. A successful job listing clears its last
error even if jobs are still unfinished, while retaining the historical failure
count. The summary's pending_run_failures counts unresolved errors, not errors
encountered only in the current collection. Failures are never converted into
successful empty records.
GitHub announced that organization retention settings will cover workflow runs created on or after October 1, 2026. The change is not retroactive. The January 2026 workflow-run and job metadata needed by this backfill was verified before collection began.