I have come across many use cases where people want to run their Lambda function for more than 5 minutes. For people who don’t know about AWS Lambda and its limitations, here’s a brief description:

AWS Lambda is a compute service that lets you run code without provisioning or managing servers. AWS Lambda executes your code only when needed and scales automatically, from a few requests per day to thousands per second. You pay only for the compute time you consume – there is no charge when your code is not running. With AWS Lambda, you can run code for virtually any type of application or backend service – all with zero administration.

Limitations:

  • Memory Allocation Range: Minimum = 128 MB / Maximum = 3008 MB (with 64 MB increments). If the maximum memory use is exceeded, function invocation will be terminated.
  • Maximum execution duration per request is 300 seconds
  • Lambda functions are short-lived, therefore they need to persist their state somewhere. Available options include using DynamoDB or RDS tables, which require fixed payments per month.
  • Lambda functions write their logs to CloudWatch, which currently is the only tool to troubleshoot and monitor your functions.

For more details about limitations click here.

WHERE I NEEDED LAMBDA TO RUN FOR MORE THAN 5 MINS

I was taking ami backups using Lambda and EC2 Tags. It was taking AMI Backups successfuly where the instances were around 20. But beyond that it used to fail despite reaching the maximum threshold. So I needed a way to run the Lambda function so that it can take AMI backup of all the instances that were there in a region, be it 20 or 200.

Then AWS SNS(Simple Notification service) came to my rescue. AWS Simple Notification Service (SNS) is a service that mainly used to push notifications/messages to large numbers of recipients like mobile device users, email recipients. But it can also be used to send messages to AWS Lambda.

Whenever a Lambda function would get triggered by anything be it CloudWatch events or cron expressions, Lambda would do it’s work and as soon as it is closer to the timeout threshold it will send notification to SNS with the required information which will be used when this Lambda function runs again. After getting the message SNS again triggers the Lambda function which is a subscriber of that SNS topic. The architecture looks like this.

The steps needed to make the above work are:

  1. Create a new SNS topic
  2. Let the Lambda function subscribe to the SNS topic
  3. Give both the Lambda function and SNS topic permission to interact with each other
  4. Use the AWS SDK (boto for Python) to send messages to AWS SNS

Here is the function which will send message to AWS SNS using boto3. Be sure to attach IAM role to the Lambda function so that it can communicate with SNS.

def publish_to_sns(message):
sns = boto3.client('sns')
return sns.publish(
TopicArn='your topic Arn',
Message=json.dumps(message),
MessageStructure='string',
MessageAttributes={
'summary': {
'StringValue': 'just a summary',
'DataType': 'String'
}
}

  • Make your lambda function “environment-aware” before timeout

We can check how much time is left by accessing context object:

  • Make event-type awared code to distinguish between Cloudwatch and SNS event

def lambda_handler(event, context):
# decide when the function should begin to send SNS message and leave
threshold_millis = 10 * 1000 # leave when there are only 10 seconds left

# some of your work here

# check remain time by accessing context object
remain_millis = context.get_remaining_time_in_millis()
if remain_millis < threshold_millis:
msg = {"necessary information for next function": "here"}
publish_to_sns(msg)
return {"partially completed": True}

return {"completed": True}

We can let our lambda function check the event content to know we’re invoked by the scheduling event or the SNS event. The reason for this check is that we may need additional information when we need to continue previous work.

A sample SNS event looks like this:

"event:": {
"Records": [
{
"EventVersion": "1.0",
"EventSubscriptionArn": "your event Arn",
"EventSource": "aws:sns",
"Sns": {
"SignatureVersion": "1",
"Timestamp": "2016-09-06T08:25:15.206Z",
"Signature": "F4Ylh6sP71lkSJIdAyKRf==",
"MessageId": "c58d6478-6de0-590c-9346-3430124b755b",
"Message": "{"necessary information for next function": "here"}",
"MessageAttributes": {
"summary": {
"Type": "String",
"Value": "just a summary"
}
}
}
}
]

The code in your Lambda function should look something like this:

if not event.get('Records', None):
# not invoked by SNS, init your job here
else:
message = json.loads(event['Records'][0]['Sns']['Message'])

# invoked by SNS, continue your previous job by using the information in the message
# which is sent by previous lambda function here.

If you just want to test whether the above mechanism work, just set the timeout in just a few seconds and send messages to the SNS immediately.

And that’s it, now you can set your lambda function timeout config to the maximum ( 5 minutes in 2016/10/01) and guarantee that it will re-invoke itself and continue to work before timeout!

Click Here to see the entire source code of the Lambda function which works on the above model to take AMI Backups and you can create your lambda functions in the same way to run more than 5 minutes.

Feel free ask any queries in the comments section below.

Leave a comment

Your email address will not be published. Required fields are marked *