Skip to content

[Brook] Committed a Workflow's stream records with its task. - #118

Closed
moetemp wants to merge 25 commits into
moe/brook/p2e-provider-pluginfrom
moe/brook/p3a-workflow-writer
Closed

moetemp wants to merge 25 commits into
moe/brook/p2e-provider-pluginfrom
moe/brook/p3a-workflow-writer

Conversation

@moetemp

@moetemp moetemp commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

What changed?

This PR adds workflow_writer(), whose records ride the completion of the activation that published them.

  • WorkflowStreamWriter has publish(value) and finish(). FINISH is idempotent per topic, and a publish after it is refused.
  • Each value is converted and hashed on the Workflow thread into one WorkflowOutputStreamCommit command. A failed activation drops it with the completion.
  • Core stages the records, writes the marker and promotes them on the accepted completion.
  • A body holds at most 1.5 MiB and an activation at most 16 MiB. The publish that crosses either raises StreamError at the call. A query or an Update validator can't publish.

Why?

A Workflow's publish becomes visible only when its task is accepted (DD-4). Core owns that commit, so lang hands over records and keeps no manifest, stage or History read (DD-87).

How did you test it?

Link to a test plan if any -

  • Unit Tests
  • Staging
  • End to End Tests

tests/contrib/streams/test_workflow_writer.py runs on the dev server and reads through Core's stream service.

moetemp added 21 commits October 9, 2026 14:58
The suite is what a provider passes to prove the publish contract, and the
memory provider is its in-process reference.
A Workflow publishes to its own stream through the provider, and a publish
from a query or an update validator is refused at the call.
Concurrent appends read the same sequence before either wrote, so records
carried wrong sequences. The provider is also marked as test-only.
# Conflicts:
#	temporalio/contrib/streams/__init__.py
A resumed read missed SUPERSEDED, a record that failed to decode was skipped or stopped every read, and a batch starting inside the newest one was accepted. The conformance suite pins the first two for every provider.
The next run after Continue-as-New can publish on the topic again, under the same empty producer id.
# Conflicts:
#	temporalio/contrib/streams/_policy.py
The writer adds each record to one output commit command on the activation's completion, so the completion encoder runs the codec on its body and Core stages and promotes it. A failed activation drops the command with the rest of its completion.
Core stages an activation's bodies in one store call, so a huge body or activation failed the task on every retry, far from the publish that caused it.
@moetemp
moetemp marked this pull request as draft October 12, 2026 06:58
@moetemp moetemp closed this Oct 13, 2026
@moetemp moetemp reopened this Oct 13, 2026
@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 13, 2026
@moetemp

moetemp commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

Replaced by Ford #164. Brook was split in two: Brook now has streams with Activity and client publish and outside readers, and Ford has the Workflow Task publish path. The code is the same, re-cut into new branches.

@moetemp moetemp closed this Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant