Repository navigation
Description
Expected Behaviour
With SqsFifoPartialProcessor(model=...), once a record fails, the remaining records (or the remaining records of the failed message group with skip_group_on_error=True) should be reported in batchItemFailures, even if one of them doesn't match the model.
Current Behaviour
The short-circuit path converts the record with the customer's model outside of any try:
if fail_entire_batch or fail_group_id:
return self.failure_handler(
record=self._to_batch_type(record, event_type=self.event_type, model=self.model),
exception=...,
)If that record fails validation, the ValidationError propagates out of process() and the whole invocation fails. The poison-pill handling from #2091 never runs here because it lives in BatchProcessor._process_record, which isn't called for short-circuited records.
As a result the partial batch response is lost and every record is retried, including the ones that were already processed successfully. With skip_group_on_error=True that includes the successful records from other message groups.
(Different code path from #8660, which is about BatchProcessor._process_record.)
Code snippet
from pydantic import BaseModel, Json
from aws_lambda_powertools.utilities.batch import SqsFifoPartialProcessor, process_partial_response
from aws_lambda_powertools.utilities.parser.models import SqsRecordModel
class OrderBody(BaseModel):
order_id: int
class Order(SqsRecordModel):
body: Json[OrderBody]
def make_record(message_id, body, group_id):
return {
"messageId": message_id,
"receiptHandle": "handle",
"body": body,
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1",
"SenderId": "sender",
"ApproximateFirstReceiveTimestamp": "1",
"MessageGroupId": group_id,
"SequenceNumber": "1",
"MessageDeduplicationId": "dedup",
},
"messageAttributes": {},
"md5OfBody": "md5",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:queue.fifo",
"awsRegion": "us-east-1",
}
def record_handler(record: Order):
if record.body.order_id == 1:
raise RuntimeError("failed to process order")
event = {
"Records": [
make_record("1", '{"order_id": 1}', "group-1"), # fails in the handler
make_record("2", '{"unexpected": 1}', "group-1"), # doesn't match the model
make_record("3", '{"order_id": 3}', "group-2"),
],
}
for skip_group_on_error in (False, True):
processor = SqsFifoPartialProcessor(model=Order, skip_group_on_error=skip_group_on_error)
# Expected: batchItemFailures for 1 and 2 (and 3 when skip_group_on_error=False)
# Actual: pydantic ValidationError for record 2 escapes and fails the invocation
print(process_partial_response(event=event, record_handler=record_handler, processor=processor))Possible Solution
The short-circuited record is never passed to the record handler, so it only needs to be convertible for failure_handler / _collect_sqs_failures, which already accept event source data classes for poison pills. Two options:
- Fall back to the event source data class when the model conversion fails (keeps
fail_messagesas model instances for valid records, same as today). - Always register short-circuited records as event source data classes. Simpler, but changes the type of entries in
fail_messagesfor customers who inspect them.
I'd go with 1 and add tests for both skip_group_on_error modes. Happy to open a PR.
Steps to Reproduce
Run the snippet above.
Powertools for AWS Lambda (Python) version
latest (develop @ 45f7a4d)
AWS Lambda function runtime
3.11
Packaging format used
PyPi
Debugging logs
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsComing soon
Activity
leandrodamascena commentedon Oct 25, 2026
Thanks @soujasK, confirmed on
developfor bothskip_group_on_errormodes. Go with option 1: fall back to the event source data class only when the model conversion fails, sofail_messageskeeps model instances for valid records. It's yours, tests for both modes please. This goes into 3.40.0.soujasK commentedon Oct 25, 2026
Thanks! PR #8663 implements option 1 with a test parametrized over both
skip_group_on_errormodes.powertools-for-aws-oss-automation commentedon Nov 10, 2026
Warning
This issue is now closed. Please be mindful that future comments are hard for our team to see.
If you need more assistance, please either reopen the issue, or open a new issue referencing this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
github-actions commentedon Nov 10, 2026
This is now released under 3.40.0 version!