[bug] v2 self-host: idle container burns ~13% CPU and writes ~190 GiB/day once bundled Motel hits its 1 GiB cap #2194

Executor version

2.0.0-beta.8 (ghcr.io/usefulsoftwareco/executor-selfhost:2.0.0-beta.8@sha256:a7d4e9d7…305a, Motel 7cbec5a). The relevant code is unchanged on v2 HEAD c935bb6 (2026-10-06).

How do you run Executor?

Self-host (Docker)

Operating system

Linux (Ubuntu, x86_64, Ryzen 9 5950X)

Integration involved

None. This happens with no traffic at all.

What happened

With no requests, the self-host container never goes idle. Measured on the workerd process:

The CPU package temperature rose about 10 °C after deploying. Almost all of the writes go to the bundled Motel store: over 30 s, Motel's *.sqlite/*.sqlite-wal were modified 42 times, the product DO's files once.

The cause is two parts that compound:

1. The idle scheduler poll is fully traced. packages/sdk/src/implementation/schedule-worker.ts ticks every pollMilliseconds: 1000 (defaultScheduleWorkerOptions). Each tick emits schedule.dispatch + FumaDB.SqlQuery.findMany + storage.query + several sql.execute spans. That is ~800 spans/min (~48k/hour) with zero user activity. In my Motel store, 97% of spans are idle polling.

2. Once Motel reaches its size cap, it runs eviction on every export. Self-host always exports to the bundled Motel when no OTLP endpoint is set (apps/hosted/self-host/src/worker.ts:117), with MOTEL_OTEL_MAX_DB_SIZE_MB: 1024 (scripts/package-runtime.ts:175). At ~48k spans/h, it reaches 1 GiB about 5–6 h after first start. In my store: used bytes 1023.98 MiB, 268k spans, and the oldest span only 5.6 h old, even though retention is 168 h.

From then on, MotelCollector.write() calls holdSizeBound after every export, which runs cleanupExpired(). That pass does:

• a SUM(span_count) over trace_summaries;
• oldest-trace selection;
• cascaded DELETEs across spans, attributes, FTS and logs;
• orphan NOT EXISTS scans and an FTS merge.

The alarm also runs this pass every 10 s. The loop stops as soon as dbSize <= maxDbSizeBytes, so it evicts down to exactly the cap, and the next export pushes it over again. Each pass may run for up to retentionPassBudgetMs (500 ms by default, not overridden for self-host). The store therefore stays pinned at the cap and pays an eviction pass per export indefinitely. That accounts for the CPU and the write volume.

There is no way to turn this off from the environment. With no OTEL_EXPORTER_OTLP_* set, the fallback to 127.0.0.1:4318 is unconditional, and the MOTEL_OTEL_* limits are baked into the generated capnp.

What you expected

An idle self-host instance should use close to 0% CPU and barely touch the disk. Local trace retention should hold roughly its configured 7 days, instead of churning through ~5 h of scheduler-poll spans.

Steps to reproduce

1. Run the beta.8 self-host image with the documented compose (no OTEL_* set), and complete setup.
2. Leave it idle for about 6 hours, until /app/motel-data/motel/*.sqlite reaches about 1 GiB.
3. Measure the workerd process: the cgroup cpu.stat delta (≈13% of a core) and /proc/<pid>/io write_bytes (≈2.2 MiB/s). Query /api/traces, or the SQLite file, to see that spans are dominated by schedule.dispatch / sql.execute / storage.query / FumaDB.SqlQuery.findMany.

Diagnostics / logs

Nothing relevant appears in the container logs. The measurements above came from /proc and a read-only snapshot of the Motel SQLite.

Possible fixes, any of which would help:

• Don't trace the idle poll. For example, emit spans only when a tick finds work, or sample it.
• Use event-driven wakeups, or back off while idle, instead of a 1 s DB poll.
• Evict with hysteresis in Motel: when over the cap, delete down to e.g. 90%. Alternatively, leave holdSizeBound to the alarm instead of running it synchronously per export.
• Add a self-host switch to disable the bundled Motel export.

Before you submit

• I searched the open issues for a duplicate.
• I removed all keys, tokens, and credentials from this report.