Skip to content

Skip no-op payload passes and context lookups without a codec or external storage - #2090

Merged
VegetarianOrc merged 4 commits into
temporalio:mainfrom
tonypodero:skip-noop-payload-passes
Nov 10, 2026
Merged

VegetarianOrc merged 4 commits into
temporalio:mainfrom
tonypodero:skip-noop-payload-passes

Conversation

@tonypodero

Copy link
Copy Markdown
Contributor

What changed

  • When the data converter has no payload codec and no external storage, workflow activations and completions skip the payload passes that cannot change a payload, and skip resolving a serialization context per command. The check that rejects external storage references without storage configured (TMPRL1105) still runs on every activation.
  • DataConverter._decode_payload_has_effect had no callers. It is now _has_codec_or_external_storage, and both the bridge and the workflow worker use it.

Why

Since 1.23 the payload passes in decode_activation / encode_completion run unconditionally (#1288, #1341). 1.25 split them into two passes per direction (#1394) and 1.35 into three (#1946). From 1.25, each visited payload sequence also asks the workflow instance for its serialization context (#1387). For a sandboxed workflow, each of those lookups is an exec into the sandbox. With the default converter, none of this work changes a payload.

We hit this after upgrading 1.22 → 1.35. Long-running workflows (3k–8k events) are rebuilt from history after each deploy, and those replays became 1.3–1.6× slower, which pushed the largest past the workflow task timeout. On three of our own histories (2.6k–7.4k events), this change brings replay time back to 1.22.0's, 25–38% less than on 1.35.0.

Replay of a synthetic history (activity + 5-minute timer in a loop, default converter, default sandbox), relative to 1.22.0. Median of 3 interleaved rounds × 5 replays; macOS arm64, Python 3.12, protobuf 5.29.6:

3,305 events 7,705 events
1.35.0 1.56× 1.51×
1.35.0 + this change 1.12× 1.15×
Where the time went (1.35.0, 7,705-event history, 1,402 activations)
  • Sandbox _run_code calls per replay: 1,406 on 1.24.0 (one activate per activation), 6,310 on 1.35.0. The extra calls are 4,203 get_serialization_context and 701 get_external_store_context.
  • decode_activation / encode_completion alone, with the default converter, per call: 0.1 / 0.1 µs on 1.22.0 and 10.9 / 12.9 µs on 1.35.0.
  • Replay vs 1.22.0 by version: 1.23 1.03–1.05×, 1.24 1.04–1.07×, 1.25 1.35–1.47×, 1.34 1.39–1.42×, 1.35 1.55–1.58×.
How it was tested
  • New test test_command_context_resolved_only_with_codec_or_external_storage: a workflow without a codec makes no context lookups, and one with a codec still does. It fails on 1.35.0 (3 lookups) and passes with this change.
  • No local Rust toolchain, so I applied the change to the 1.35.0 wheel (these three files match main apart from unrelated lines) and ran the 1.35.0 tests against it:
    • test_serialization_context.py, test_extstore.py and worker/test_extstore.py: 88 passed, the same as unpatched.
    • worker/test_workflow.py, worker/test_replayer.py, worker/test_activity.py, and test_serialization_context.py with the new test: 348 passed, 2 skipped, 0 failed.
  • Not run locally: poe lint-types and the full poe test on main.

@tonypodero
tonypodero requested a review from a team as a code owner November 5, 2026 15:23
@CLAassistant

CLAassistant commented Nov 5, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@tonypodero
tonypodero force-pushed the skip-noop-payload-passes branch from 9bc9d39 to c1a34f7 Compare November 6, 2026 07:25

@jmaeagle99 jmaeagle99 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looking good. Asking for a change to preserve an existing behavior and a test adjustment.

Comment thread temporalio/worker/_workflow.py Resolved
Comment thread tests/test_serialization_context.py Resolved
…ut external storage

- With a cached workflow and no codec or external storage, activation
  failures are converted with the context-free converter, as before.
- encode_completion runs the external storage pass only when external
  storage is configured, so a codec-only worker makes no store-context
  lookups.
- The test also counts get_external_store_context lookups and covers a
  converter with external storage.

@jmaeagle99 jmaeagle99 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I'll defer to @tconley1428 for any other adjustments and to decide when to merge.

@VegetarianOrc
VegetarianOrc merged commit cef4726 into temporalio:main Nov 10, 2026
33 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants