How to build powerful back-ends easily with Serverless

Published May 16, 2017
How to build powerful back-ends easily with Serverless

Building an image processor on AWS Lambda

Originally published on Medium

I started working as a Software Developer Full-time in 2012. You build it, you run it was a core value of our small team. But, I knew nothing about servers. I soon had to learn about subnets, load balancers, database clusters and more to support the code I wrote. For me, building software that solves problems is fun, managing servers isn’t.

In early 2016 I attended an AWS meetup. At this meetup, Sam Kroonenburg spoke about how he built A Cloud Guru with a Serverless Architecture on AWS. It blew me away. The idea of using AWS Lambda to build web apps without managing or scaling servers had me hooked.

Since then I have dedicated my spare time to learning and blogging about Serverless. Six months after that meetup Sam offered me a job and I now build Serverless systems Full-time for A Cloud Guru.

Learning about Serverless has had a big impact on the way I build software. Serverless gained a lot of momentum and maturity in 2016. Perhaps you looked at it last year but didn’t try it or maybe you’ve never heard of it. Whatever the case, there has never been a better time to get started.

What is (and isn’t) Serverless?

There is still a lot of debate on what is Serverless. Right now the answer depends on who you ask. There are two questions, what is a Serverless product and what is a Serverless architecture.

A Serverless product is fully managed with granular scaling and granular billing. Products include but are not limited to:

Note I did not include any Platform as a Service (PaaS) products like Amazon RDS, Heroku or Google App Engine. For these services, you pay per hour and scale by adding or removing underlying servers. You might not have to manage servers but you still need to think about them.

So what is a Serverless architecture? I define a Serverless Architecture as:

An event driven system that utilises FaaS and other fully managed services for logic and persistence.

If you are still scratching your head don’t worry. This will make more sense once you see an example.

Getting started with Serverless

The best way to learn is to practice. Let’s pretend that a client has asked me to reinvent anonymising faces on images.

Instead of blurring a face, my client wants to replace each face with an emoji. The emoji must reflect the emotion of that face.

To deliver this service I need to:

  1. Allow my client to upload images
  2. Detect the faces in each image
  3. Determine the emotion of each face
  4. Replace each face with an emoji
  5. Store the edited image

Instead of building it all myself, I am going to use a Serverless architecture on AWS.

Serverless Architectures

There are three services I need to develop this system.

Using those services I developed the following architecture.

architecture.png

Let’s go back to the definition of a Serverless architecture.

An event driven system that utilises FaaS and other fully managed services for logic and persistence.

The architecture I have designed fits this definition.

  1. When a user uploads a file an ObjectCreated event is produced and a Lambda function is invoked.
  2. The Lambda function calls Amazon Rekognition to detect the faces and emotion of each face in the uploaded image.
  3. The Lambda function processes the image and persists the image in Amazon S3

The simplicity of this design is what makes Serverless compelling.

What languages and tools can I use

If you want to build this system yourself, you need to create the infrastructure and code.

To develop the code you can use Node.js, Python, Java, or C#. There is an AWS SDK for each of these languages. If your language is missing, watch for language support to grow over 2017.

To create the infrastructure there are a few options. The first option is to use the AWS console. This is a good way to start while you are learning. But this is not recommended for production services. The other option is to develop your infrastructure as code. Two good tools for this are CloudFormation or Terraform.

To simplify your experience I recommend using a tool that helps you deploy your infrastructure and your code. The Serverless Framework is the tool that I recommend using. It is an open source project with a great community of contributors (including me). Other alternatives include Apex, Claudia, Sparta and more.

Show me the code

You’ve seen the architecture, let’s look at some code. I’m going to use the Serverless Framework and NodeJs.

The first step is to define the infrastructure using the Serverless Framework. To do this I created a new service

serverless create --template aws-nodejs --path emoticon-faceswap

Infrastructure

This creates a folder containing a configuration file named serverless.yml. The functions and infrastructure for my service is defined here.

service: serverless-emoticon-faceswap
provider:
  name: aws
  runtime: nodejs4.3
  # Allow Lambda to access Rekognition and S3
  iamRoleStatements:
    - Effect: Allow
      Action:
        - rekognition:DetectFaces
      Resource: '*'
    - Effect: Allow
      Action:
        - s3:GetObject
        - s3:PutObject
      Resource: arn:aws:s3:::${self:custom.bucketName}/*
custom:
  bucketName: ${self:service}-${opt:stage}-uploads
functions:
  # Define my Lambda function
  faceswap:
    handler: src/faceswap.handler
    timeout: 30
    environment:
      BUCKET_NAME: ${self:custom.bucketName}
      ALLOWED_EXTENSIONS: .jpg|.jpeg|.png
      PROCESSED_DIR_NAME: processed
    events:
     # Create a S3 bucket that triggers my Lambda function
      - s3:
          bucket: ${self:custom.bucketName}
          event: s3:ObjectCreated:*
          rules:
            - prefix: uploads/

This configuration creates my faceswap Lambda function and S3 bucket. The Lambda function will be invoked when an object is created in the uploads/ folder.

Code

Every Lambda function has a handler that serves as the entry point. I set the handler for my Lambda function to src/faceswap.handler in serverless.yml. This is a combination of the file path src/faceswap.js and the function to invoke in that file handler.
Each handler function has three parameters:

module.exports.handler = (event, context, callback) => {
   ...
   
   // Use callback() and return information to the caller.  
}
  • Event: Data relating to the event. In this case it will contain information about the file that was created in S3
  • Context: Lambda runtime information
  • Callback: A normal node callback that you can use to return an error or result.

For my project the Lambda function needs to perform three tasks.

  1. Getting images from the S3 event
  2. Call the Rekognition API
  3. Process the images and save the results.
'use strict';

const BbPromise = require('bluebird');
const path = require('path');

const logger = require('./logger');
const processor = require('./processor');
const rekognition = require('./rekognition');

const BUCKET_NAME = process.env.BUCKET_NAME;
const ALLOWED_EXTENSIONS = process.env.ALLOWED_EXTENSIONS.split('|');

// Process S3 Event Object
const getImagesFromEvent = (event) => event.Records.reduce((accum, r) => {
    if (r.s3.bucket.name === BUCKET_NAME) {
        const key = r.s3.object.key;
        const extension = path.extname(key).toLowerCase();

        if (ALLOWED_EXTENSIONS.indexOf(extension) !== -1) {
            accum.push(key);
        }
    }

    return accum;
}, []);

// Handler function that is invoked by Lambda
module.exports.handler = BbPromise.coroutine(function* (event, context, cb) {
    try {
        logger.log('Recieved Event', event);
        const images = getImagesFromEvent(event);

        logger.log('Found images on event', images);
        const imageFaces = yield rekognition.detectFacesOnImages(images);

        logger.log('Detected faces', imageFaces);
        yield processor.processImages(imageFaces);

        cb(null);
    } catch (err) {
        logger.log('Error', err);
        cb(err);
    }
});

You can view the code that calls Amazon Rekognition and processes the images on GitHub.

The final step is to deploy the project and test it. To deploy the project I ran the following command.

sls deploy --stage your_stage_name

I then uploaded images from Pexels to S3 and checked the results.

results.png

This was a really fun project to build. Building a project project like this is the best way to learn. It will help you understand event driven architectures and how to create Serverless systems.

Where can I learn more?

There are lots great resources available for you to learn more about Serverless.

I hope post has inspired you to look at Serverless. There are lots of great use cases to try. A HTTP API, Chatbot or Alexa Skill are just a few projects to try.

If you have any questions about this project or Serverless in general you can contact me on Medium or Twitter.

Discover and read more posts from John McKim
get started
Enjoy this post?

Leave a like and comment for John

10
4
4Replies
Aniruddha Diwakar
19 days ago

Nice description.

Ben Yeh
4 months ago

Awesome post, thanks for sharing. Just curious: where do you keep your production environment variables?
Having a convenient way to create and update environment variables is important. And it’s probably not a good a idea to save them in a local folder.
I used to store my credentials and token in DynamoDB, but that’s still far from being a very convenient and secure solution.

Aniruddha Diwakar
19 days ago

you can use ‘dotenv’ to process environment variables.

Ben Yeh
19 days ago

dotenv is good for development, but I don’t think it’s a good idea to store security data in your repo.

Show more replies

Get curated posts in your inbox

Read more posts to become a better developer