In the past few months, I wanted to explore Serverless Architecture and I wanted to build a serverless app.
After hearing all the good things about serverless. I was interested in exploring and understanding about the caveats attached to the serverless architecture.
I had so many questions… Is Serverless Architecture best for application development? When to choose it? and what I need to consider before building a serverless application.
After searching the internet I came across aws blog post with the title “Resize Images on the Fly with Amazon S3, AWS Lambda, and Amazon API Gateway” - Blog Link
Though the above blog did not consider the limitations I later found out. It gave me a starting point.
I felt excited and wanted to add a front end to this system where user can upload the image to a website and resize it based on their desired height and width.
something similar to the famous resizer out there.: imageresizer.com
My First Attempt (novice):
Build an API and call it in the front end to resize the image using AWS Lambda and give ability to the user to download the image after resizing.
Front end:
Vanilla JS, Custom CSS, Javascript
Back end:
AWS Chalice, Python, Lambda, Pillow
For Building API I used
For Resizing I used Pillow Library
Road Blocks in this attempt:
Problem1:
Image and Dimensions in POST req body using Lambda Proxy bad Idea! Pillow library couldn’t recognize the image to resize cause the Image was disturbed(data changed) by intermediate nodes before reaching the lambda.
Why did this happen?
So, AWS Chalice is built on top of Lambda proxy integration that doesn’t handle multipart post requests with images very well. It handles text very well separately but not both!
Lambda Proxy Problem - When I tried to submit the image using a post request from the front end. The image is converted into binary and the binary format goes through several intermediary nodes before reaching the required resizing lambda.
These intermediate nodes add extra information that change the image data(binary format) when I try to mix and send the dimension data (height and width) with image data in the POST - request Body. It added other lambda proxy information to the binary image which led to the image being undetectable here.
Solution:
So, I decided to send the POST request with image data and dimensions information in the POST request URL instead of the POST body.
Example: <Lambdafunctionurl>:/HeightXWidth/
To make sure the image is received by lambda correctly. I decided to use some kind of parser.
Thanks to contributors of the Python library - multipart parser, I found what I need is a multipart parser which can parse the image from streaming binary/base64 formats to required jpg, png formats and can separate any text information present in POST request into separate dictionaries in python.
After using the parser I was able to resize the image using Pillow Library but now I ran into: The Lambda timeouts :-(
The Lambda timeouts
The default time-out limit is 1min which can be increased to 15mins for a lambda function.
My Second Attempt (present)
Problems in Attempt1:
The Lambda timeouts
So you shouldn’t be doing everything in one go that is: you can’t upload, resize the image, and send the resized image all at once This will exceed the Lambda time out and cause an error.
Max Invocation Payload Limitation: 6MB
Limitations of Request and Response Size in lambda: In Serverless architecture, we are also limited with a max payload limit (6MB). so images larger than 5-6MB will fail if submitted in one fetch request.
Solution:
Then, I decided to approach the task of uploading and resizing in two phases. Upload Phase: Created an upload function in the front end and back end that can accept the image in parts without exceeding the time limits of AWS Lambda. I used s3 to store the images and improved the front end and backend code to accept the image in parts at the backend by first saving the part of the object in s3 generating a unique ID using UUID Library and combining the image in subsequent requests. Resize Phase:
For the Resizing phase, I created the API to resize the image using the unique ID of the image that is generated by the backend system and saved in the front end after the uploading request is complete. to additional resizing requests that may arise for the same image in the front end.
Example:
<Lambdafunctionurl>:/<ResizeRequest>/
<ImageObjectURL>/HeightXWidth/
I wanted to revamp the UI at the front end and adapt React for the front end to make it more responsive this time and for advanced state management. Now Stack looks like this:
Front end:
React, BootStrap,
Javascript functions
Back end:
Python Based: AWS Chalice, AWS S3
UUID, Pillow Library
Present Project:
you might ask me Is it perfect? no, it is not but it helped me understand the several limitations caused by serverless architecture that you might not think of when building a dedicated application on an EC2 server or on-premise using popular backend programming like Python’s Django or Java’s Spring. Limitation:
Update Oct 1 - You can now resize up to 4K resolution (3840 x 2160). To keep the costs of running lambda low i capped the resize rate until 4K.
Sept 29- Can’t request a resizing request with more than 1920px X 1920px resolution this exceeds the 6MB payload limit for response. I will try to improve this in a few weeks. It should work very well for general requests like for social media resizing.
Conclusion: So Is serverless bad? - Not actually, it is very good for applications that need to scale.
See other amazing benefits below.
Benefits of AWS Lambda:
No servers to manage
Continuous scaling
Millisecond metering
Increases innovation
So it depends on your unique use case, you can choose serverless or traditional dedicated server architecture. If you are considering serverless architecture, consider separating and dividing your requests and responses into smaller requests according to the 6MB limit and you need to be aware of the execution limitations of aws lambda. It works very well when building Apps that don’t require large payload sizes or payloads that can be broken down into smaller requests.
Summary:
Serverless Image Resizer
Developed a Serverless image resizing service using leveraging AWS Lambda, API Gateway, and Amazon S3
Enabled users to upload images and dynamically resize them with desired dimensions on-demand. Implemented Chunked uploads in the frontend and backend to bypass the 6MB payload limit of AWS lambda.
Generated prisigned URLs for users to share their resized images privately.