Skip to content

Conversation

@shaked-lokits
Copy link

Issue number: closes #5362

Summary

Add support for processing Kafka and MSK events with partial batch failure reporting, leveraging the new ReportBatchItemFailures functionality that AWS recently enabled for Kafka event sources.

Changes

  • Add EventType.Kafka to batch processor for unified MSK/self-managed Kafka support
  • Implement _collect_kafka_failures() for composite identifiers (topic-partition + offset)
  • Add _get_records_from_event() helper to flatten Kafka's nested record structure
  • Add KafkaItemIdentifier TypedDict for type safety
  • Add 14 functional tests covering sync/async processing, partial failures, and edge cases
  • Update batch processing documentation with Kafka section and SAM template
  • Update Kafka utility documentation with cross-reference to batch processing

User experience

Before: Users had to manually implement partial batch failure handling for Kafka/MSK events, including constructing the correct response format with composite identifiers.

After: Users can use the familiar batch processing API:

from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, process_partial_response

processor = BatchProcessor(event_type=EventType.Kafka)

def record_handler(record):
    # Process record
    return record.value

def lambda_handler(event, context):
    return process_partial_response(event=event, record_handler=record_handler, processor=processor, context=context)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@shaked-lokits shaked-lokits requested a review from a team as a code owner January 12, 2026 16:13
@boring-cyborg
Copy link

boring-cyborg bot commented Jan 12, 2026

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@boring-cyborg boring-cyborg bot added documentation Improvements or additions to documentation tests labels Jan 12, 2026
@pull-request-size pull-request-size bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jan 12, 2026
@shaked-lokits shaked-lokits force-pushed the feat/batch-kafka-support branch from c7c227c to e9661cd Compare January 12, 2026 16:14
@sonarqubecloud
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: add msk to batch processing

1 participant