Skip to content

Commit d8e4358

Browse files
serdakasaciserdakasaci
committed
commit sqs lamdba lmi with provisioned mode esm
1 parent aaed990 commit d8e4358

4 files changed

Lines changed: 563 additions & 0 deletions

File tree

‎sqs-lambda-lmi-esm-provisioned-sam/README.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Amazon SQS + Lambda Managed Instances (LMI) with Provisioned Mode ESM
2+
3+
This pattern deploys an Amazon SQS Standard Queue connected to an AWS Lambda function via an Event Source Mapping (ESM) configured with **Provisioned Mode** — an ESM feature that pre-allocates dedicated polling resources for predictable, high-throughput message processing.
4+
5+
Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/sqs-lambda-lmi-esm-provisioned-sam](https://serverlessland.com/patterns/sqs-lambda-lmi-esm-provisioned-sam)
6+
7+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
8+
9+
## Requirements
10+
11+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
12+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
13+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
14+
* [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed
15+
16+
## Build Instructions
17+
18+
1. From the command line, use AWS SAM to build the AWS resources for the pattern as specified in the template.yml file:
19+
```
20+
sam build
21+
```
22+
23+
## Deployment Instructions
24+
25+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
26+
```
27+
git clone https://github.com/aws-samples/serverless-patterns
28+
```
29+
2. Change directory to the pattern directory:
30+
```
31+
cd sqs-lambda-lmi-esm-provisioned-sam
32+
```
33+
1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file:
34+
```
35+
sam deploy --guided
36+
```
37+
1. During the prompts:
38+
* Enter a stack name
39+
* Enter the desired AWS Region
40+
* Accept the default parameter values or tune them for your workload
41+
* Allow SAM CLI to create IAM roles with the required permissions.
42+
43+
Once you have run `sam deploy --guided` mode once and saved arguments to a configuration file (samconfig.toml), you can use `sam deploy` in future to use these defaults.
44+
45+
1. Note the outputs from the SAM deployment process. These contain the resource names and/or ARNs which are used for testing.
46+
47+
## How it works
48+
49+
1. A producer (any AWS service, SDK client, or CLI) sends messages to the SQS Standard Queue.
50+
2. The ESM's provisioned event pollers continuously long-poll the queue using up to 10 SQS API calls per second per poller.
51+
3. When messages are available, pollers assemble batches (up to `BatchSize` messages, waiting up to `MaxBatchingWindowInSeconds`) and invoke the Lambda function concurrently.
52+
4. Lambda scales the number of active pollers between `MinimumPollers` and `MaximumPollers` based on queue depth, adding up to **1,000 concurrent executions per minute**.
53+
5. If a message fails processing after 3 attempts (`maxReceiveCount`), it is moved to the Dead Letter Queue.
54+
6. A CloudWatch Alarm fires as soon as any message lands in the DLQ.
55+
56+
## Testing
57+
58+
1. Get the queue URL from the stack outputs:
59+
60+
```bash
61+
aws cloudformation describe-stacks \
62+
--stack-name <your-stack-name> \
63+
--query "Stacks[0].Outputs"
64+
```
65+
66+
2. Send test messages:
67+
68+
```bash
69+
QUEUE_URL=<QueueUrl from outputs>
70+
71+
# Send a single message
72+
aws sqs send-message \
73+
--queue-url $QUEUE_URL \
74+
--message-body '{"orderId": "123", "amount": 99.99}'
75+
76+
# Send a batch of 10
77+
for i in $(seq 1 10); do
78+
aws sqs send-message \
79+
--queue-url $QUEUE_URL \
80+
--message-body "{\"orderId\": \"$i\", \"amount\": $((RANDOM % 100))}"
81+
done
82+
```
83+
84+
3. Check Lambda logs:
85+
86+
```bash
87+
sam logs --stack-name <your-stack-name> --tail
88+
```
89+
90+
4. Inspect the ESM status and poller count:
91+
92+
```bash
93+
ESM_ID=<EventSourceMappingId from outputs>
94+
95+
aws lambda get-event-source-mapping --uuid $ESM_ID
96+
```
97+
98+
## Cleanup
99+
100+
1. Delete the stack
101+
```bash
102+
aws cloudformation delete-stack --stack-name STACK_NAME
103+
```
104+
2. Confirm the stack has been deleted
105+
```bash
106+
aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'STACK_NAME')].StackStatus"
107+
```
108+
----
109+
Copyright 2025 Amazon.com, Inc. or its affiliates. All Rights Reserved.
110+
111+
SPDX-License-Identifier: MIT-0

‎sqs-lambda-lmi-esm-provisioned-sam/example-pattern.json‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
{
2+
"title": "Amazon SQS to AWS Lambda with Lambda Managed Instances (LMI) Provisioned Mode ESM",
3+
"description": "This pattern shows how to process Amazon SQS messages using AWS Lambda Managed Instances (LMI) with a Provisioned Mode Event Source Mapping for faster scaling and higher throughput.",
4+
"language": "Python",
5+
"level": "300",
6+
"framework": "AWS SAM",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"A producer sends messages to an Amazon SQS Standard Queue. The Event Source Mapping (ESM) is configured with Provisioned Mode, which pre-allocates dedicated event pollers between a configurable minimum and maximum. Each poller handles up to 1 MB/s throughput, 10 concurrent Lambda invocations, and 10 SQS API calls per second.",
11+
"The Lambda function runs on Lambda Managed Instances (LMI) — an EC2-backed execution model that supports multiconcurrency, allowing each execution environment to handle multiple SQS batches simultaneously. LMI instances run inside a private VPC with outbound access via a NAT Gateway.",
12+
"Partial batch failure reporting (ReportBatchItemFailures) ensures only failed messages are returned to the queue. After 3 failed attempts, messages are moved to a Dead Letter Queue. A CloudWatch Alarm fires as soon as any message lands in the DLQ.",
13+
"This pattern deploys one SQS Standard Queue, one Dead Letter Queue, one Lambda function with an LMI Capacity Provider, one Event Source Mapping with Provisioned Mode, a VPC with private subnets and NAT Gateway, and one CloudWatch Alarm."
14+
]
15+
},
16+
"gitHub": {
17+
"template": {
18+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/sqs-lambda-lmi-esm-provisioned-sam",
19+
"templateURL": "serverless-patterns/sqs-lambda-lmi-esm-provisioned-sam",
20+
"projectFolder": "sqs-lambda-lmi-esm-provisioned-sam",
21+
"templateFile": "template.yaml"
22+
}
23+
},
24+
"resources": {
25+
"bullets": [
26+
{
27+
"text": "Using Lambda with Amazon SQS",
28+
"link": "https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html"
29+
},
30+
{
31+
"text": "Lambda Managed Instances (LMI) overview",
32+
"link": "https://docs.aws.amazon.com/lambda/latest/dg/managed-instances.html"
33+
},
34+
{
35+
"text": "Event Source Mapping Provisioned Mode for SQS",
36+
"link": "https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#sqs-esm-provisioned-mode"
37+
},
38+
{
39+
"text": "Reporting batch item failures for SQS",
40+
"link": "https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting"
41+
}
42+
]
43+
},
44+
"deploy": {
45+
"text": [
46+
"sam build",
47+
"sam deploy --guided"
48+
]
49+
},
50+
"testing": {
51+
"text": [
52+
"See the GitHub repo for detailed testing instructions."
53+
]
54+
},
55+
"cleanup": {
56+
"text": [
57+
"Delete the stack: <code>aws cloudformation delete-stack --stack-name STACK_NAME</code>."
58+
]
59+
},
60+
"authors": [
61+
{
62+
"name": "Serda Kasaci Yildirim",
63+
"image": "https://drive.google.com/file/d/1rzVS1hrIMdqy6P9i7-o7OBLNc0xY0FVB/view?usp=sharing",
64+
"bio": "Serda is a Solutions Architect at Amazon Web Services (AWS) based in Vienna with 3 years at AWS, specializing in serverless technologies and Event-Driven Architecture.",
65+
"linkedin": "serdakasaci"
66+
}
67+
]
68+
}

‎sqs-lambda-lmi-esm-provisioned-sam/src/handler.py‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
import json
2+
import logging
3+
4+
logger = logging.getLogger()
5+
logger.setLevel(logging.INFO)
6+
7+
8+
def lambda_handler(event, context):
9+
batch_item_failures = []
10+
11+
for record in event.get("Records", []):
12+
try:
13+
body = json.loads(record["body"])
14+
logger.info("Processing message", extra={"message_id": record["messageId"], "body": body})
15+
except Exception as e:
16+
logger.error("Failed to process record %s: %s", record["messageId"], e)
17+
batch_item_failures.append({"itemIdentifier": record["messageId"]})
18+
19+
return {"batchItemFailures": batch_item_failures}

0 commit comments

Comments
 (0)