Skip to content

fix(parser): accept plain-text string payloads - #8766

Closed
zhanfan-818 wants to merge 1 commit into
aws-powertools:developfrom
zhanfan-818:fix/parser-plain-text-payloads
Closed

zhanfan-818 wants to merge 1 commit into
aws-powertools:developfrom
zhanfan-818:fix/parser-plain-text-payloads

Conversation

@zhanfan-818

@zhanfan-818 zhanfan-818 commented Nov 4, 2026 •

Copy link
Copy Markdown

Issue number: closes #8695

Summary

Changes

CloudWatchLogsEnvelope parses every logEvents[*].message through _parse_and_validate_event, which for str payloads called adapter.validate_json(data) unconditionally (aws_lambda_powertools/utilities/parser/functions.py:74). CloudWatch Logs subscriptions forward every event in the log group, including Lambda runtime lines such as START RequestId: ..., which are plain text rather than JSON.

This change keeps validate_json as the first attempt and falls back to validate_python when it raises ValidationError; if the fallback also fails, the original json_invalid error is re-raised so callers that expect ValidationError keep the same error and message:

        except ValidationError as exc:
            try:
                return adapter.validate_python(data)
            except ValidationError:
                raise exc from None

Nothing changes for JSON payloads ("message": json.dumps(...)), which still take the fast path.

User experience

Before, a subscription that included any runtime log line failed for the whole batch:

pydantic_core._pydantic_core.ValidationError: 1 validation error for str
  Invalid JSON: expected value at line 1 column 1 [type=json_invalid,
  input_value='START RequestId: eadc91c8-... Version: $LATEST\n', input_type=str]

After, with model=str, plain-text messages are returned verbatim and JSON messages are still parsed into the model:

RESULT: OK parsed=1 value='START RequestId: eadc91c8-... Version: $LATEST\n'

Verification

Reproduced and verified locally on develop (3b26155) with Python 3.13.9 and the pydantic==2.14.2 pinned by uv.lock — the report mentions 3.14.10, which I did not have available.

Repro script builds a real awslogs payload (base64 + zlib) whose single message is a START ... line and calls parse(..., model=str, envelope=CloudWatchLogsEnvelope):

$ python repro.py                      # before
RESULT: EXCEPTION   (ValidationError: Invalid JSON)

$ python repro.py                      # after
RESULT: OK parsed=1 value='START RequestId: eadc91c8-... Version: $LATEST\n'

Focused tests, including the two regression tests added in this PR:

$ python -m pytest tests/unit/parser/_pydantic/test_cloudwatch.py \
    tests/unit/parser/_pydantic/test_cloudwatch_non_json_messages.py -q
8 passed, 1 warning in 0.70s

The new file covers one plain-text message and a mix of JSON and plain-text messages; the existing six tests in test_cloudwatch.py still pass unchanged, including the ones that assert ValidationError for non-conforming payloads.

I used an AI assistant to draft this patch; I reviewed it, ran the reproduction and the test command above myself, and the output quoted here is from those runs.

Two notes from reading the code:

  • The report points at envelopes/functions.py; the isinstance(data, str) branch is in utilities/parser/functions.py:71-74, reached from envelopes/base.py:44.
  • The two tests named in the report (test_cloudwatch_envelope_handles_non_json_messages, test_cloudwatch_envelope_handles_mixed_json_and_plain_text) are not in the repository, so they are added here.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

CloudWatch Logs subscriptions forward Lambda runtime lines such as
"START RequestId: ...", which are not JSON. _parse_and_validate_event
called validate_json unconditionally for str payloads and raised
ValidationError before the model could see the value.

Try validate_json first, then fall back to validate_python, and re-raise
the original error when the fallback fails so existing callers keep the
same error semantics.

Adds regression tests for a single plain-text message and for a mix of
JSON and plain-text messages.
@zhanfan-818
zhanfan-818 requested a review from a team as a code owner November 4, 2026 04:40
@boring-cyborg

boring-cyborg Bot commented Nov 4, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@boring-cyborg boring-cyborg Bot added the tests label Nov 4, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 4, 2026
@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

@zhanfan-818

Copy link
Copy Markdown
Author

Closing as a duplicate: #8699 landed the same fallback (validate_json → validate_python) and merged yesterday, so the parser change this branch carries is already on develop. No point spending reviewer time on it.

The two regression tests here (single plain-text message, and a mix of JSON and plain-text logEvents) may still be worth having if #8699 did not add them — happy to reopen a tests-only PR against develop if a maintainer wants that; otherwise I will leave it alone.

@zhanfan-818 zhanfan-818 closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: CloudWatchLogsEnvelope raises ValidationError on non-JSON logEvents[*].message (e.g. Lambda START/END/REPORT)

1 participant