Skip to content

Latest commit

 

History

History

github-actions-queue

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 

README.md

GitHub Actions queue data

This centrally executed workflow collects one compact timing record for every terminal GitHub Actions job in active public repositories under open-telemetry. Records are written to the orphan otelbot/github-actions-queue-data branch in this repository and can be queried to find queue-time outliers.

Data writers always retain only the latest parentless Git snapshot of the data branch. All historical job measurements in the current files, collection checkpoints, and report inputs remain intact.

The snapshot protocol is unconditional and no longer needs DATA_BRANCH_SNAPSHOTS. The obsolete repository variable can be deleted after the workflow environment cleanup is merged. See the shared storage requirements. The hourly collection and Pages deployment schedules do not change.

Queue time is measured per job:

queue_seconds = job_started_at - job_created_at

Matrix jobs are separate records. Jobs that never receive a runner or have a negative queue interval are not stored because they have no usable runner queue time. This excludes skipped jobs, jobs cancelled before assignment, and check runs created only to publish results.

When only failed jobs are rerun, GitHub also returns cloned records for jobs that did not execute again. The clones have a later created_at but retain the original execution timestamps, producing a negative queue interval that the collector discards.

Dashboard

The hourly queue-time dashboard is published at https://open-telemetry.github.io/shared-workflows/github-actions-queue/. It opens to the latest seven days of GitHub-hosted runner data and provides:

  • Required GitHub-hosted or self-hosted runner selection
  • Runner labels scoped to the selected host category
  • Repository filtering
  • Inclusive UTC start and end dates that default to the latest seven days
  • Shareable URLs that retain the selected filters
  • Hourly p50, p90, p95, and p99 queue times in UTC
  • Exact p50, p90, p95, and p99 queue times across the selected range

The self-hosted category matches runner labels case-insensitively against:

self-hosted
cncf-*
oracle-*
*-s390x

These broad patterns cover the special self-hosted runners documented in the OpenTelemetry community asset inventory. All other assigned runners are classified as GitHub-hosted. In particular, otel-windows-latest-8-cores is GitHub-hosted even though it uses an organization-defined runner group.

The collector incrementally updates report-state.json.gz and daily report partitions under report-data/ on the data branch. The Pages artifact includes only the dashboard assets and those derived partitions, not the raw job files or collector checkpoints.

Repository administrators must enable GitHub Pages once in the repository settings and select GitHub Actions as the source. The deployment uses the standard github-pages environment and needs no new GitHub App permission, token, or secret.

GitHub App

Use the private organization-owned GitHub App named OpenTelemetry Actions Telemetry. Its homepage is https://github.com/open-telemetry/shared-workflows, and it has no webhook or event subscriptions.

The App requires only:

  • Actions: read-only
  • Metadata: read-only, granted automatically

Install it for All repositories so one installation token can read Actions data across the organization and new repositories are covered automatically. It requests no webhook events and no write permissions. The collector lists only public repositories and rejects an explicitly selected repository unless GitHub reports private: false.

Store its credentials in the protected environment of shared-workflows:

  • Variable ACTIONS_TELEMETRY_CLIENT_ID
  • Secret ACTIONS_TELEMETRY_PRIVATE_KEY

The workflow creates a short-lived installation token for API reads. Its built-in GITHUB_TOKEN, scoped to shared-workflows, writes the data branch.

Collection

.github/workflows/github-actions-queue-collector.yml runs at minute 17 of every hour. It discovers active public repositories dynamically, then processes closed one-hour workflow-run windows. A repository checkpoint lets the next run resume an incomplete window without re-emitting repositories already committed.

The collector prioritizes live data, then uses the remaining run budget to backfill from 2026-09-17T02:00:00Z toward 2026-01-01T00:00:00Z. Backfill processes newer hours first and continues automatically on each schedule. backfill-state.json stores the oldest complete hour plus partial repository progress, independently of the live state.json cursor.

Up to four repositories are processed concurrently. Workflow invocations stop after at most 20 total windows, 12,000 requests, or 40 minutes of collection. These limits leave room below the App's 15,000-request hourly quota and the workflow's 50-minute timeout. A stopped run commits its records and checkpoints before the next schedule resumes it.

Runs that have not reached a terminal state are saved in state.json. Later collections revisit them and emit their jobs only after the parent run is completed and every runner-assigned job is terminal. Runnerless jobs are excluded before this completion check, so they do not block measurable sibling jobs. Completed runs containing only runnerless jobs leave the pending set without emitting records. A job reported as in_progress or queued is also terminal when GitHub supplies both a completion timestamp and a terminal conclusion. Such assigned jobs missing either piece of evidence still block the entire run. Records retain GitHub's reported job status and conclusion. Job listing uses filter=all, so actual executions from every available attempt are retained when the run is finalized. Carry-forward clones created by partial reruns are not stored.

Each live or backfill collection selects up to 25 saved pending runs before collecting windows. Five slots are reserved for unresolved API failures and 20 for other unfinished runs; unused slots are shared. Both groups use the oldest last-attempt time, falling back to creation time for unattempted runs. Selected runs are processed in that same chronological order, within the request, runtime, and REST quota limits. Backfill retries run only after live collection catches up without pausing and leaves a window budget.

The collector tracks the installation's REST quota. If it exhausts the quota, it commits the partial checkpoint and resumes during the next scheduled run.

The completed runnerless and negative-queue cleanup is recorded in migrations/remove-unmeasurable-jobs-v1.json on the data branch.

GitHub limits a filtered workflow-run search to 1,000 results. The collector splits busy one-hour windows into smaller ranges until each result can be paginated safely. Repository and workflow-run pages contain up to 100 results. Job pages contain up to 50 results to reduce GitHub jobs-API server errors on large pages. The collector checks that all pages yield the reported number of unique jobs before finalizing a run.

Data layout

Each successful collection adds one immutable gzip-compressed JSON Lines file:

jobs/date=2026-09-15/collection-123456789-1.jsonl.gz

state.json and backfill-state.json are mutable collector checkpoints, not reporting data. Each job line contains:

Field Meaning
schema_version Job-record schema version.
organization GitHub organization, currently open-telemetry.
repository Repository name under open-telemetry.
repository_id Stable GitHub repository ID when the runs API supplies it.
workflow_name, workflow_id Workflow identity.
run_id, run_attempt, run_created_at Workflow run identity and creation time.
event, from_fork Trigger and whether the head repository was a fork. from_fork is null when GitHub no longer provides the head repository.
head_branch, head_sha Source revision.
job_id, job_name Individual job identity. Matrix values normally appear in job_name.
job_status, job_conclusion GitHub's reported state and result. A terminal job can retain a stale in_progress or queued status.
job_created_at, job_started_at, job_completed_at GitHub job timestamps.
queue_seconds Start minus creation time.
runner_assigned Always true for retained queue records.
runner_labels, runner_name, runner_group_name Runner classification.
html_url Direct link to the job.
collected_at Time the collector finalized the record.

The format deliberately retains self-hosted jobs, actual retries, fork runs, and large queue values. Reports should filter those dimensions rather than discarding raw records during collection.

Local use

Set GH_TOKEN, GITHUB_AUTH_TOKEN, or GITHUB_TOKEN, then use a temporary state and output directory:

python3 .github/scripts/github-actions-queue/collect.py \
  --repository shared-workflows \
  --start 2026-09-15T00:00:00Z \
  --state /tmp/actions-queue-state.json \
  --output-dir /tmp/actions-queue-jobs \
  --collection-id local-test \
  --max-windows 1

The command prints a JSON summary containing the request count plus starting and ending limit, remaining, and reset values. The starting snapshot comes from the first ordinary API response because GitHub's /rate_limit response can lag the bucket reported by repository endpoints. The summary also reports the lowest remaining fraction observed because GitHub can return fluctuating quota headers across sequential public-repository requests.

The summary's exclusions contains runnerless_jobs, runnerless_runs, and negative_queue_jobs counts across live and backfill collection, including pending retries. runnerless_runs counts completed, nonempty runs containing only runnerless jobs. Counts cover finalized runs in checkpointed repository windows and resolved pending retries, not runs still waiting for assigned jobs or discarded partial repository work.

Known REST limitation

The runs API filters by original creation time, not the time of a later rerun. The collector preserves all attempts returned when it finalizes a discovered run, but a rerun requested after that run leaves the pending set can fall outside later creation-time windows. Closing this gap may require a bounded lookback or workflow_job webhook ingestion.

Failed job lookups are retained in state.json with their failure count, last error, and last attempt time. Backfill failures are retained in backfill-state.json the same way. Later collections retry them while continuing to discover other runs. The workflow emits a warning as long as any failed lookup remains unresolved. A successful job listing clears its last error even if jobs are still unfinished, while retaining the historical failure count. The summary's pending_run_failures counts unresolved errors, not errors encountered only in the current collection. Failures are never converted into successful empty records.

GitHub announced that organization retention settings will cover workflow runs created on or after October 1, 2026. The change is not retroactive. The January 2026 workflow-run and job metadata needed by this backfill was verified before collection began.