Skip to content

create OTel tracer provider once per process - #2041

Merged
tconley1428 merged 1 commit into
temporalio:mainfrom
avanish-garg:fix/lambda-otel-warm-invocations
Nov 9, 2026
Merged

tconley1428 merged 1 commit into
temporalio:mainfrom
avanish-garg:fix/lambda-otel-warm-invocations

Conversation

@avanish-garg

Copy link
Copy Markdown
Contributor

What was changed

The provider-building code in lambda_worker/otel.py moved into _ensure_tracer_provider, which apply_defaults now calls. It installs the global replay-safe TracerProvider once and returns early on later calls while that provider is still the global one. If set_tracer_provider did not take (another provider already owns the global slot), the unused provider is shut down immediately. The apply_defaults docstring now says the provider is created once per process. Adds a changelog/fixed fragment. apply_tracing is unchanged (it builds a fresh plugin per invocation and its flush hook looks up the global provider each time).

The diff looks bigger than the change: most of it is the existing provider-building block moved into the helper. Logic and warning text are unchanged; a cold start behaves exactly as before.

Why?

run_worker calls configure on every Lambda invocation, so apply_defaults built a new TracerProvider plus OTLP BatchSpanProcessor (daemon thread and gRPC channel) each time. OpenTelemetry only allows the global provider to be set once, so from the second warm invocation the new provider was never installed or shut down, and a thread and channel leaked per invocation.

Trade-offs: the first call wins, so endpoint and service name are fixed for the process lifetime (they come from env or OtelOptions, static per container). The module-level cache has no lock, as configure runs sequentially in the main thread. A foreign global provider still triggers OpenTelemetry's "Overriding" warning on each call; avoiding that would mean skipping provider creation whenever a non-proxy provider is already global, which is a behavior change I left out.

Checklist

  1. Closes [Bug] lambda_worker otel.apply_defaults leaks a span exporter thread and gRPC channel on every warm invocation #2040

  2. How was this tested:
    New tests in tests/contrib/aws/lambda_worker/test_otel.py (class TestApplyDefaultsWarmInvocations):

  • test_reuses_provider_across_invocations: the exporter is built once across several apply_defaults calls.
  • test_does_not_warn_about_overriding_provider
  • test_unused_provider_is_shut_down_when_global_already_set
  • test_flush_hook_exports_on_every_invocation: spans are still exported on warm invocations 2 and 3.

Without the change 3 of these fail and 67 other tests in tests/contrib/aws/lambda_worker pass; with it all 70 pass. The reproduction from the issue now shows 1 export thread after each of 5 invocations (it grew 1, 2, 3, 4, 5 before). Ruff, pyright, basedpyright and mypy were run on the changed files only. Not run: the full lint task, the rest of the suite, a real Lambda or a real OTLP collector (the tests use fakes and a recording exporter). The new tests reset OpenTelemetry's global provider through trace._TRACER_PROVIDER and _TRACER_PROVIDER_SET_ONCE via monkeypatch, as there is no public reset API.

  1. Any docs updates needed?
    No. Docstring updated. The README still mentions apply_metrics where the function is build_metrics_telemetry_config; I left that out of this PR.

apply_defaults is called from the configure callback, which run_worker
invokes on every Lambda invocation. Each call built a new
TracerProvider with an OTLP BatchSpanProcessor, but OpenTelemetry only
allows the global provider to be set once, so from the second warm
invocation on the new provider was never installed or shut down,
leaking an export thread and gRPC channel per invocation.

Install the provider once and reuse it on later calls. If another
provider already owns the global slot, shut the unused one down.

@johanngan johanngan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix @avanish-garg, LGTM, just one small comment suggestion.



class TestApplyDefaultsWarmInvocations:
"""``configure`` runs on every Lambda invocation in the same warm process."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment doesn't really describe the test suite itself. I'd suggest something like

Suggested change
"""``configure`` runs on every Lambda invocation in the same warm process."""
"""Test repeated apply_defaults calls per process, simulating warm invocations"""

@tconley1428
tconley1428 merged commit ce519ce into temporalio:main Nov 9, 2026
20 checks passed
patbeqo added a commit to temporalio/skill-temporal-observability that referenced this pull request Nov 14, 2026
Add a serverless OTel reference (Lambda and Cloud Run helpers, collector
sidecar, shutdown, runtime and once-per-process provider notes) linked
from SKILL.md.

Serverless worker operators need telemetry defaults that survive
short-lived invocations.

Upstream:
- temporalio/sdk-dotnet#721
- temporalio/sdk-dotnet#844
- temporalio/sdk-go#2588
-
https://github.com/temporalio/sdk-go/releases/tag/contrib/opentelemetry/otlpworker/v0.1.0
-
https://github.com/temporalio/sdk-go/releases/tag/contrib/gcp/cloudrun/otel/v0.1.0
- temporalio/sdk-java#2901
- temporalio/sdk-java#3022
- temporalio/sdk-java#3064
- temporalio/sdk-python#1767
- temporalio/sdk-python#2041

Eval: Knowledge task, Claude Code, Sonnet 5.5
Baseline: 6/7 items, 230k tokens, 64s
Improved: 7/7 items, 57k tokens, 18s
Delta: +1 item, -75% tokens, -72% time

Eval task: temporalio/skill-evals#427

---------

Co-authored-by: temporal-plugin-updater[bot] <3389934+temporal-plugin-updater[bot]@users.noreply.github.com>
Co-authored-by: Patrik Beqo <patbeqo@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] lambda_worker otel.apply_defaults leaks a span exporter thread and gRPC channel on every warm invocation

3 participants