How to use Amazon S3 Object Lambda to generate thumbnails

Did you ever run into a scenario where you wanted to transform objects retrieved from Amazon S3 “on the fly”? For example, to convert a file to a different format, or exclude sensitive data, or resize, compress and transform an image?

Then you’ll be happy to hear that recently added Amazon S3 Object Lambda offers exactly that, a capability to modify the output of a standard S3 GET request.

I couldn’t stop myself from trying out this new feature and, as a result, wrote this step by step guide to use S3 Object Lambda to generate thumbnail images for an imaginary gallery🐣. Hopefully it will help you understand how to use S3 Object Lambda and save some time in overcoming couple of challenges I run into.

What do we want to do

We want an easy way to create and retrieve image thumbnails.

A thumbnail is a smaller copy of an image, for example, used for a gallery. There are different ways how we can create and retrieve thumbnails. We can store different sizes of the same image and retrieve the size we need. But this takes extra storage space and lacks flexibility if you want to add a new image dimension.

Another option would be to shrink the image on the client side, but this is also far from ideal. It will take unnecessary bandwidth to download the original image. Moreover, images resized by the browser often have visual artefacts.

What would be great, is to store only original images and transform them upon retrieval to the size we need. That’s where S3 Object Lambda will help us.

Let’s create a project to demonstrate how we can resize images stored in S3 bucket with the help of Amazon Lambda and S3 Object Lambda.

The plan 🧭

Let’s look at what we need to do, step by step:

1. Create an S3 bucket and put there original images
2. Set up an Access Point for the bucket we created
3. Create an empty Lambda function
4. Add Object Lambda Access Point to connect S3 bucket and Lambda function
5. Create a script to retrieve an original image from S3 bucket and its thumbnail from the S3 Object Lambda
6. Add code to Lambda function to resize an image
7. Run the script created in step 5 to get two images or different sizes

For Lambda function code I’ll be using Javascript and NodeJS. If you are more familiar with Python I left couple of links with more examples I found at the end of this article.

Having a plan is halfway to success 😉 Now let’s dive into it!

Instructions 🛠️

Before anything else, you’ll need to have an AWS account, if you still don’t have it. You can find more information in this article in the knowledge center.

Also, for the testing script we’ll be using NodeJs and AWS SDK for Javascript and we’ll rely on AWS CLI to create a lambda layer with up-to-date SDK, but it can be done through the console as well.

Step1: S3 bucket

We need a place where we store original images.

In the console go to Amazon S3 and create a new bucket. I’ve left all default settings in place when creating a new bucket.

Once created, add couple of images there. I uploaded three .jpeg images of my fluffy neighbours🐓.

Step2: S3 Access Point

Next we’ll need to set up an Access Point for our bucket. In S3 console go to the bucket you created, then go to Access Points and click on Create access point.

Apart from setting a name I left all properties to be default ones.

By the end of this step you should have an S3 bucket with several images there and a created access point.

Step3: Lambda function

Next, let’s work on the “brains” of our image transformation. First, create a new Lambda function

I created a function from scratch and used Node.js 14.x as a runtime, for the rest I left default settings

The Lambda function would need to have permissions to access s3 bucket as well as be able to call method writeGetObjectResponse to return a transformed result.

For this we’ll need to add additional policies to the execution role. You’ll find the execution role under Configuration settings.

I’ve added a new policy with following permissions

Before we add actual code to our lambda (where I had to go through couple of iterations), let’s add Object Lambda Access Point.

Step4: Object Lambda Access Point

You can find Object Lambda Access Point in the menu of S3 console. Go there and you’ll see a button to create a new Object Lambda Access Point.

Give it a name and specify the S3 Access Point we created in step 2. You can do it by clicking on Browse S3 or entering ARN of the Access Point.

Next, choose your lambda function

The rest of the settings I left as default. When you are done, click Create Object Lambda Access Point.

Now, we have a basic setup with all pieces in place. We’re still lacking transformation code in the Lambda function, but before adding it let’s write a test script to request images and test S3 Object Lambda we created .

Step 5: Script to download an original image and its thumbnail

I’ll use AWS SDK for Javascript for this task . To start, create a simple project with

npm init

And run

npm install aws-sdk

Next, in this project add an index.js file.

In its main function we retrieve and save two images, one from S3 bucket (the original image), and second from S3 Object Lambda (the thumbnail).

Since we haven’t add proper code to the lambda function, you’ll get an error when running this script

The Lambda exited without successfully calling WriteGetObjectResponse

Indeed, to make it work we need to utilise WriteGetObjectResponse to transform the response. Let’s do it in the next step.

Step6: Add code in Lambda function

In the lambda function we want to do the following:

• get original image from S3 bucket with the help of axios
• use sharp to resize it (we’ll talk below how to add nodeJs packages)
• use properties from event’s context to generate a new request
• call writeGetObjectResponse and submit resized image

Here is an initial version of the handler that I wrote:

Initial code of lambda function, which won’t work

To my frustration, this didn’t work. There is nothing wrong with the code above. In fact, it might perfectly work in the future. But at the moment of writing this article the setup has couple of challenges.

Challenge # 1 : writeGetObjectResponse is not defined

When calling client.writeGetObjectResponse, we’ll get an error, so deeply familiar to those of us who use Javascript — “Undefined is not a function”. Strange, you might think, according to S3 documentation, that’s a valid function.

The reason is simple —S3 Object Lambda is a brand new feature and AWS Lambda execution environment is not yet up to date with latest AWS SDK. And this is completely normal. The AWS SDK versions built into the Lambda execution environment aren’t always up-to-date. That’s easy to fix though!

First, how do we verify that the version of SDK is indeed the problem?

We can compare the SDK version in Lambda environment to the one which supports the feature we want to use. To do so in Lambda code we can return (or console.log) the version, as in the snippet below.

And to find the SDK version which supports writeGetObjectResponse I checked the release notes in githab repository for aws-sdk-js, and found this json file.

And indeed the SDK version used in Lambda execution environment was 2.804.0, while the one which has writeGetObjectResponse was added only in 2.867.0.

But nothing to be worried about. We can add latest SDK Version as a new layer! A detailed instruction can be found in knowledge-center and I recommend following it, it makes steps simple.

In short, what I did was creating a package on my computer where I installed latest version of SDK and run AWS CLI command to publish a layer version. If you don’t want to use CLI, you can upload a package for a new layer by using the console. It is also described in the same article in the knowledge-center.

Once you have done that, go to the lambda function you created and add a new layer.

You can select newly created layer either by specifying its ARN or by selecting it from the list of custom layers, as I did below

Since I am using two more libraries (axios and sharp), I added them to the same layer, so when following the instructions not only I run npm install aws-sdk, but I did the same for axios and sharp.

You know that moment when you expect that now all should work? It didn’t 😅

Challenge#2 : MethodNotAllowed

When I run the lambda function again I got a new error — 405: MethodNotAllowed: The specified method is not allowed against this resource.

After verifying that all proper permissions are in place and playing around I found the issue — the endpoint and service name are not resolved correctly for the WriteGetObjectResponse API. And while SDK team is working on fixing it, we can use the following workaround to set correct endpoint and its prefix.

So, after the adjustments this is how the code of lambda function looks like:

Lambda function with a workaround for 405 error

We’re almost done! But there was one more thing which I had to adjust.

Challenge#3 : darwin-x64' binaries cannot be used on the ‘linux-x64’ platform

When calling a method to resize the image, I got a new error in Lambda logs that darwin-x64' binaries cannot be used on the ‘linux-x64’ platform. By googling I found this conversation in stackoverflow.

To fix it I revisited the package I created for Lambda layer. In particular I removed sharp package completely and run the command below to install it with specified the platform

npm install --arch=x64 --platform=linux sharp

Next, updated the layer and pointed my lambda function to use its latest version. This resolved the issue.

Step 7 : Run the script

Now, if we run a small script that we created in step 5, we’ll successfully get two images!

Conclusions

We have created S3 Object Lambda which transforms original images into a smaller thumbnail. Now, we don’t need to store different size versions of our image and can transform an image “on the fly” when retrieving it from the bucket.

More information and more examples

Some useful resources which I found while experimenting with the feature

• https://aws.amazon.com/s3/features/object-lambda — overview of S3 Object Lambda and scenarios where it is useful
• https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/ — a guide how to use S3 Object Lambda to transform text using Python
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/transforming-objects.html — instructions for transforming objects with S3 Object Lambda with code in Java. There are useful notes on how to configure IAM policies as well.
• https://eoins.medium.com/using-s3-object-lambdas-to-generate-and-transform-on-the-fly-874b0f27fb84 — another guide with an example in Python