Skip to content

[Bridge] Defined the stream service in IDL. - #130

Closed
moetemp wants to merge 107 commits into
moe/ford/w8-workflow-demosfrom
moe/bridge/n0a-stream-service-idl
Closed

moetemp wants to merge 107 commits into
moe/ford/w8-workflow-demosfrom
moe/bridge/n0a-stream-service-idl

Conversation

@moetemp

@moetemp moetemp commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

What changed?

This PR defines the stream service temporal.sdk.streams.v1.TemporalStreams in IDL and generates its Python bindings.

  • temporal_streams.nexusrpc.yaml has a sync append of one batch and a long-poll read from a cursor.
  • A reader carries Core's read state in ReadInput.state, so the handler holds nothing (DD-99). A record too large to cross comes as RecordWire.error at its cursor.
  • The contract is Core's copy byte for byte, and a test pins its digest. The next PR generates from Core's copy and removes this one.
  • scripts/gen_nexus_streams_api.py installs the moetemp nexgen fork at 47fc2c0c until the fixes land upstream. CI runs its --check.

Why?

A surface that crosses a process boundary is defined once in IDL and generated (DD-36). A process that can't reach the store reads and writes through a Nexus endpoint instead.

How did you test it?

Link to a test plan if any -

  • Unit Tests
  • Staging
  • End to End Tests

tests/contrib/streams/test_nexus_service_idl.py, and gen_nexus_streams_api.py --check on a fresh install of the pin.

moetemp added 30 commits October 9, 2026 14:54
Streams store records as temporal.sdk.streams.v1.StreamRecord. Only the
Python SDK reads it, so it is generated here rather than in the API repo.
Records map to the envelope proto in one place, and the body helpers hash
the converted payload before the codec so a retry deduplicates.
Errors tell a refused write from one with an unknown outcome, and expired
cursors from invalid ones. A cursor names its stream, so a provider can
refuse one from another stream.
A topic is defined once with its record type, and a StreamRef names a
Workflow's stream as data another process can open.
A provider hands out handles and producers, and registers once on a client
or Worker, so the accessors never pick between two stores silently.
The suite is what a provider passes to prove the publish contract, and the
memory provider is its in-process reference.
A ref that names an owner kind from a later release now reaches the ref's
own kind check when decoded, instead of failing the converter's type check.
The generated v1 modules are rewritten on every proto run, so the warning
goes on the hand-written package above them.
Only provider code uses them, so exporting them would commit the package
to an interface no application needs.
# Conflicts:
#	temporalio/contrib/streams/__init__.py
#	tests/contrib/streams/test_records_wire.py
A caller must tell a write the store refused from a store that failed,
and the store's own exceptions must not leak, so both get a StreamError
family that the specific errors join.
# Conflicts:
#	temporalio/contrib/streams/__init__.py
Only providers resolve a topic argument, so applications do not need it
and the package should not commit to it.
Providers must serialize calls on one producer, and a caller needs to know
what to do after a cancelled or unknown append.
Concurrent appends read the same sequence before either wrote, so records
carried wrong sequences. The provider is also marked as test-only.
The contract is the one source for every SDK's stream service surface, so the Python models, service definition and HTTP caller are generated from it. Regeneration needs the pinned nexgen fork until its annotations merge upstream.
The store compares bodies as they arrive, so a writer whose codec uses a fresh nonce has to keep its encoded batch for the retry.
The fork pin moves to the nexgen revision with done-field, so read_until_records returns once a read says the stream ended instead of polling to its deadline.
The pinned nexgen now marks docstring code as reStructuredText literals and no longer re-exports the mapped StreamRef, so poe gen-docs no longer fails on unresolved links. The package takes StreamRef from the SDK.
A stored kind at 3 would decode as SUPERSEDED, which readers refuse. The format test now pins field types and labels too, since a retyped field breaks stored records.
A resumed read started with no attempts seen, so a retry right after the cursor reached the reader as plain DATA with no SUPERSEDED.
Skipping it with a log warning lost data the reader never heard of. The error carries the cursor, so a caller can resume past it on purpose.
Core asks about owners over the connection that connected the store, so a second connection got the first one's answers. A Worker now needs a client made with Client.connect whatever connected the store first, and the delete test checks a stream that held a record.
A read that resumes after a cursor sees an earlier attempt only through the store's reader, so the answer has to carry that supersession. done also covers a closed stream, and the record wording is the final one, so N15 needn't touch the contract.
The Core review fixes add the topic flag to the append script, keep stages on a NotFound completion and cap reads. Only proto comments and the envelope's Ruby package changed, so the regenerated files differ in docstrings.
C5b 5ae0ae71 only explains why a refused promotion is retried, so the final pin matches the Brook Core top.
Core serves the contract, so a reader carries Core's read state and the handler holds none, and the token schema joins the contract. The copy now matches Core's canonical one, generated with nexgen at G18.
The contract copy must be Core's own until the SDK generates from vendored Core, so a digest pins it. Core's token vectors now leave out an unpinned run_id, and the shared file covers every case.
Grepping the source missed a line moved above the early return and failed on a harmless rename. A stub transport checks what each request carries.
The token decoder now refuses what Core refuses, and the long poll keeps its state when an answer carries none.
CN4 8835dc5a says how large a read state may be, and CN5 b0ac595e adds a vector for DEL in an id. The copies must stay Core's own bytes.
@moetemp
moetemp marked this pull request as draft October 12, 2026 06:58
Merged with the old Brook base 1aa69e3. The result is w8 plus n0a's own change.
@moetemp
moetemp changed the base branch from moe/brook/p2e-provider-plugin to moe/ford/w8-workflow-demos October 13, 2026 20:36
@moetemp moetemp mentioned this pull request Oct 14, 2026
1 of 3 tasks
@moetemp

moetemp commented Oct 14, 2026

Copy link
Copy Markdown
Owner Author

Replaced by Bridge v2 #182. Bridge was re-cut onto Jetty (stream wake-ups in one namespace) with no Ford content. The same-namespace notifier parts moved to Jetty, and the Nexus parts are in Bridge v2.

@moetemp moetemp closed this Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant