Generate Python Lambda Layers with a Dockerfile

This blog post assume you have an AWS Account operational !

We will focus on Lambda functions running Python but keep in mind that Lambda can run many different languages (Node, Java, Ruby etc).

Why am I here?

• I want to remove my "modules not found" error in my AWS Lambda functions
• I want to understand basics of Lambda Layers
• I want to ZIP my Python modules in my own docker environment (using a Dockerfile)

Quick recap of AWS Lambda and Lambda Layers

Lambda is a FAAS for "Function As A Service", this service allows you to run code without worrying about servers. Autoscaling is self-managed. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

Lambda layers provide an easy way to package libraries and dependencies that you can use across multiples Lambda functions. Using layers reduces the size of uploaded deployment archives (here your package libraries and dependencies) and makes it faster to deploy your code to AWS.

Lambda layers are .zip archives.

Using a Dockerfile for creating the archive

There are different ways to create your zip archive, the reasons of using a Dockerfile here are the benefits of Docker (Running on all platforms) and the simplicity to share, custom and improve the Dockerfile.

FROM python:3.8-slim-buster
LABEL maintainer="julienQNN"

# Install dependencies

RUN apt update && apt install zip -y && mkdir -p /layer/python/lib/python3.8/site-packages && rm -rf /var/lib/apt/lists/*

# Copying your requirements.txt file in the container

COPY requirements.txt ./

# install packages, zip and copy to the root folder

RUN pip install --no-cache-dir -r requirements.txt -t layer/python/lib/python3.8/site-packages/ && cd /layer && zip -r lambdalayerpackage.zip *

Don't forget to put your dependencies in a requirements.txt file at the root folder with the Dockerfile like in this example :

For generating the requirements.txt file, you can use pipreqs at the root of your Python project.

Notice that i'm using the official python:3.8-slim-buster Docker image here, however you can use the 3.9 or 3.10 versions, depending on your needs.

At the root folder :

1. Build your Docker image with the DockerFile :

docker build -t {NameOfYourImage} .
# E.g. docker build -t myFirstLambdaLayersImage .

2. List your images (you should see the one you just created) :

docker images

3. Run your newly created image :

docker run -tid --name {container name you want} {name or id of the image}
# E.g. docker run -tid --name PythonLayersContainer myLambdaLayersImage .

Apologize for this terrible naming 🙃

Using the archive for your Lambda layers

At this point, you should have a lambdalayerpackage.zip file in the same folder of your DockerFile. What you need to do next is importing this zip archive in AWS. There are many ways to do so, for example we could do it with a CLI and sending it directly to Amazon S3 or in a pipeline, anyway we will keep this simple and visual b using the AWS Console.

1. Connect to AWS Lambda
2. Choose the AWS region you need in the top right corner of the console (the layers has to be in the same regions of your functions to wrap them)
3. Click on Layers in the Additional Resources sub menu
4. Click on Create a layers buttons
5. Fill in the fields, choose your Python runtime version and import your local zip file

Now if you go back in Layers you should see you're ready to use Lambda Layer.

7. Click on Functions and choose your function
8. Click on Add a layer and choose the custom layer you created before

Tada ! Every time your function will run, the python packages from the layer will be used.