Skip to content

fix(kafka): read Confluent Protobuf message indexes as zigzag varints - #5960

Merged
dreamorosi merged 3 commits into
aws-powertools:mainfrom
Manishram-ai:fix/kafka-protobuf-message-indexes
Nov 14, 2026
Merged

dreamorosi merged 3 commits into
aws-powertools:mainfrom
Manishram-ai:fix/kafka-protobuf-message-indexes

Conversation

@Manishram-ai

@Manishram-ai Manishram-ai commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Kafka Protobuf deserializer read the Confluent message index length as a plain varint and then skipped that many bytes. For any message type other than the first one in the schema, that cut into the message, and records decoded with fields missing and no error. It now reads the indexes as zigzag varints, which is what Confluent writes, the same as the Python fix in aws-powertools/powertools-lambda-python#8670.

Changes

  • In packages/kafka/src/deserializer/protobuf.ts, the length and each index are read with sint32() before decoding. The int32 attempt and the shared varintOrder swap are gone, so a record no longer decodes differently depending on earlier records.
  • A negative index count now throws Invalid Confluent message index array length, like Python does, and the Glue and Confluent paths share one catch.
  • deserializer.protobuf.test.ts has a new test for message indexes [0], [1], [2], [1, 0] and [70] (two bytes), and one for a negative count. [1], [2] and the negative count fail on main, and they all pass with this change.
  • The Confluent fixture test in consumer.test.ts now checks the full decoded record for all four records, with userId, name and email added to its parserSchema. On main it fails when run by itself.
  • The "handles protobuf messages with delimiters" test used 02 01 00, the old plain-varint form of [1, 0]. I changed it to the zigzag form 04 02 00.
  • npm run test:unit:coverage, npm run lint:ci, npm run build:tests and npm run build pass in packages/kafka, and coverage stays at 100%.

The old plain-varint index format isn't supported anymore. Data written that way doesn't throw, it decodes wrong: 02 01 00 plus the Product payload comes back as {"price":999.99}. confluent-kafka doesn't write that format anymore, and the Python fix dropped it too.

Issue number: closes #5959


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@dreamorosi dreamorosi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround @Manishram-ai. The core fix is right: reading the count and each index as zigzag varints matches the Confluent wire format and aws-powertools/powertools-lambda-python#8670, and dropping the varintOrder swap gets rid of the cross-record state. I checked the bytes in the new it.each cases by hand, and tests and coverage pass locally.

Two things need changing before merge, both inline:

  1. Reject a negative index count, like Python does.
  2. Assert more than userId in the complex Confluent test.

One more for the PR description: please give the old plain-varint format its own line, and say what happens to it now. Old-format data doesn't throw, it decodes wrong. 02 01 00 + the Product payload comes back as {"price":999.99}. That's probably fine since confluent-kafka stopped writing it, but anyone reading the changelog should know.

The two comments marked nit are optional.

@dreamorosi dreamorosi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this!

@dreamorosi
dreamorosi enabled auto-merge (squash) November 14, 2026 10:43
@dreamorosi
dreamorosi merged commit 8a6dfc2 into aws-powertools:main Nov 14, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L PRs between 100-499 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Kafka Protobuf deserializer reads Confluent message indexes as plain varints

2 participants