Repository navigation
fix(kafka): read Confluent Protobuf message indexes as zigzag varints - #5960
dreamorosi merged 3 commits into
Conversation
dreamorosi
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround @Manishram-ai. The core fix is right: reading the count and each index as zigzag varints matches the Confluent wire format and aws-powertools/powertools-lambda-python#8670, and dropping the varintOrder swap gets rid of the cross-record state. I checked the bytes in the new it.each cases by hand, and tests and coverage pass locally.
Two things need changing before merge, both inline:
- Reject a negative index count, like Python does.
- Assert more than
userIdin the complex Confluent test.
One more for the PR description: please give the old plain-varint format its own line, and say what happens to it now. Old-format data doesn't throw, it decodes wrong. 02 01 00 + the Product payload comes back as {"price":999.99}. That's probably fine since confluent-kafka stopped writing it, but anyone reading the changelog should know.
The two comments marked nit are optional.
Summary
The Kafka Protobuf deserializer read the Confluent message index length as a plain varint and then skipped that many bytes. For any message type other than the first one in the schema, that cut into the message, and records decoded with fields missing and no error. It now reads the indexes as zigzag varints, which is what Confluent writes, the same as the Python fix in aws-powertools/powertools-lambda-python#8670.
Changes
packages/kafka/src/deserializer/protobuf.ts, the length and each index are read withsint32()before decoding. Theint32attempt and the sharedvarintOrderswap are gone, so a record no longer decodes differently depending on earlier records.Invalid Confluent message index array length, like Python does, and the Glue and Confluent paths share onecatch.deserializer.protobuf.test.tshas a new test for message indexes[0],[1],[2],[1, 0]and[70](two bytes), and one for a negative count.[1],[2]and the negative count fail onmain, and they all pass with this change.consumer.test.tsnow checks the full decoded record for all four records, withuserId,nameandemailadded to itsparserSchema. Onmainit fails when run by itself.02 01 00, the old plain-varint form of[1, 0]. I changed it to the zigzag form04 02 00.npm run test:unit:coverage,npm run lint:ci,npm run build:testsandnpm run buildpass inpackages/kafka, and coverage stays at 100%.The old plain-varint index format isn't supported anymore. Data written that way doesn't throw, it decodes wrong:
02 01 00plus theProductpayload comes back as{"price":999.99}. confluent-kafka doesn't write that format anymore, and the Python fix dropped it too.Issue number: closes #5959
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.