Building Enterprise Level Web Applications on AWS Lambda with the DEEP Framework

Apr 14, 2016 ~ 11 min read

Since the beginning, Mitoc Group has been building web applications for enterprise customers. We are a small group of developers who are helping customers with their entire web development process, from conception through execution and down to maintenance.

Overview

Since the beginning, Mitoc Group has been building web applications for enterprise customers. We are a small group of developers who are helping customers with their entire web development process, from conception through execution and down to maintenance. Being in the business of doing everything is very hard, and it would be impossible without using AWS foundational services, but we incrementally needed more. That is why we became earlier adopters of serverless approach and developed an ecosystem called Digital Enterprise End-to-end Platform (shortly DEEP) with AWS Lambda at core.

In this post, we will dive deeper into how DEEP is leveraging AWS Lambda to empower developers build cloud-native applications or platforms using microservices architecture. We will walk through the thinking process of identifying the front-end, the back-end and the data tiers required to build web applications with AWS Lambda at core. We will focus on the structure of AWS Lambda functions we use, as well as security, performance and benchmarking steps that we take to build enterprise-level web applications.

Enterprise-level Web Applications

Our approach to web development is full-stack and user-driven, focused on UI (aka User Interaction) and UX (aka User eXperience). But before going into the details, we'd like to emphasize the strategical (biased and opinionated) decisions we have made early:

• We don't say "no" to customers; Every problem is seriously evaluated and sometimes we offer options that involve our direct competitors
• We are developers and we focus only on the application level; Everything else (platform level and infrastructure level) must be managed by AWS
• We focus our 20% of effort to solve 80% of work load; Everything must be automated and pushed on the services side rather than ours (clients)

To be honest and fair, it doesn't work all the time as expected, but it does help us to learn fast and move quickly, sustainably and incrementally solving business problems through technical solutions that really matters. But the definition of "really matters" is different from customer to customer, quite unique in some cases. Nevertheless, what we learn from our customers is that enterprise-level web applications must provide the following 7 common expectations:

1. Be secure — security through obscurity (e.g. Amazon IAM, Amazon Cognito);
2. Be compliant — governance-focused, audit-friendly service features with applicable compliance or audit standards;
3. Be reliable — Service Level Agreements (e.g. Amazon S3, Amazon CloudFront);
4. Be performant — studies show that page loads longer than 2s start impacting the users behavior;
5. Be pluggable — successful enterprise ecosystem is mainly driven by fully integrated web applications inside organizations;
6. Be cost-efficient — benefit of AWS Free Tier, as well as pay only for services that you use and when you use them;
7. Be scalable — serverless approach relies on abstracted services that are pre-scaled to AWS size, whatever that would be.

Architecture

This article will describe how we have transformed a self-managed task management application (aka todo app) in minutes. The original version can be seen on www.todomvc.com and the original code can be downloaded from https://github.com/tastejs/todomvc/tree/master/examples/angularjs. The architecture of every web application we build or transform, including the one described above, is similar to the reference architecture of the realtime voting application published recently by AWS on Github:

The todo app is written in AngularJS and deployed on Amazon S3, behind Amazon CloudFront (the front-end tier). The tasks management is processed by AWS Lambda, optionally behind Amazon API Gateway (the back-end tier). The tasks metadata is stored in Amazon DynamoDB (the data tier). The transformed todo app, along with instructions on how to install and deploy this web application, is described in this blog post and the code is available on Github:

In this article, we will focus on AWS Lambda functions and the value proposition it offers to us and our customers.

AWS Lambda Functions

Let's get into the details of the thinking process and the AWS Lambda functions that we have written for this web app. The goal of the todo app is to manage tasks in a self-service mode. End users can view tasks, create new tasks, mark or unmark a task as done, and clear completed tasks. From UI and UX point of view, that leads us to 4 user interactions that will require 4 different back-end calls:

1. web service that retrieves task(s)
2. web service that creates task(s)
3. web service that deletes task(s)
4. web service that updates task(s)

Simple reorder of the above identified back-end calls leads us to basic CRUD (Create, Retrieve, Update, Delete) operations on the Task data object. And these are the simple logical steps that we take to identify the front-end, the back-end and the data tiers of (drums beating, trumpets playing) our approach to microservices, which we prefer to call microapplications.

Therefore, coming back to AWS Lambda, we have written 4 small node.js functions that are context-bounded and self-sustained (each below microservice corresponds to the above identified back-end web service):

'use strict';

import DeepFramework from 'deep-framework';

export default class Handler extends DeepFramework.Core.AWS.Lambda.Runtime {
  /**
   * @param {Array} args
   */
  constructor(...args) {
    super(...args);
  }

  /**
   * @param request
   */
  handle(request) {
    let taskId = request.getParam('Id');

    if (taskId) {
      this.retrieveTask(taskId, (task) => {
        return this.createResponse(task).send();
      });
    } else {
      this.retrieveAllTasks((result) => {
        return this.createResponse(result).send();
      });
    }
  }

  /**
   * @param {Function} callback
   */
  retrieveAllTasks(callback) {
    let TaskModel = this.kernel.get('db').get('Task');

    TaskModel.findAll((err, task) => {
      if (err) {
        throw new DeepFramework.Core.Exception.DatabaseOperationException(err);
      }

      return callback(task.Items);
    });
  }

  /**
   * @param {String} taskId
   * @param {Function} callback
   */
  retrieveTask(taskId, callback) {
    let TaskModel = this.kernel.get('db').get('Task');

    TaskModel.findOneById(taskId, (err, task) => {
      if (err) {
        throw new DeepFramework.Core.Exception.DatabaseOperationException(err);
      }

      return callback(task ? task.get() : null);
    });
  }
}

'use strict';

import DeepFramework from 'deep-framework';

export default class extends DeepFramework.Core.AWS.Lambda.Runtime {
  /**
   * @param {Array} args
   */
  constructor(...args) {
    super(...args);
  }

  /**
   * @param request
   */
  handle(request) {
    let TaskModel = this.kernel.get('db').get('Task');

    TaskModel.createItem(request.data, (err, task) => {
      if (err) {
        throw new DeepFramework.Core.Exception.DatabaseOperationException(err);
      }

      return this.createResponse(task.get()).send();
    });
  }
}

'use strict';

import DeepFramework from 'deep-framework';

export default class Handler extends DeepFramework.Core.AWS.Lambda.Runtime {
  /**
   * @param {Array} args
   */
  constructor(...args) {
    super(...args);
  }

  /**
   * @param request
   */
  handle(request) {
    let taskId = request.getParam('Id');

    if (typeof taskId !== 'string') {
      throw new InvalidArgumentException(taskId, 'string');
    }

    let TaskModel = this.kernel.get('db').get('Task');

    TaskModel.updateItem(taskId, request.data, (err, task) => {
      if (err) {
        throw new DeepFramework.Core.Exception.DatabaseOperationException(err);
      }

      return this.createResponse(task.get()).send();
    });
  }
}

'use strict';

import DeepFramework from 'deep-framework';

export default class extends DeepFramework.Core.AWS.Lambda.Runtime {
  /**
   * @param {Array} args
   */
  constructor(...args) {
    super(...args);
  }

  /**
   * @param request
   */
  handle(request) {
    let taskId = request.getParam('Id');

    if (typeof taskId !== 'string') {
      throw new DeepFramework.Core.Exception.InvalidArgumentException(taskId, 'string');
    }

    let TaskModel = this.kernel.get('db').get('Task');

    TaskModel.deleteById(taskId, (err) => {
      if (err) {
        throw new DeepFramework.Core.Exception.DatabaseOperationException(err);
      }

      return this.createResponse({}).send();
    });
  }
}

Each above file with related dependencies is compressed into .zip file and uploaded to AWS Lambda. If you're new to this process, we'd strongly recommend to follow How to Create, Upload and Invoke an AWS Lambda function tutorial.

Back to our 4 small node.js functions, you can see that we have adopted ES6 (aka ES2015) as our coding standard. And we are importing deep-framework in every function. What is this framework anyway and why we're using it everywhere?

Full-stack Web Framework

Let us step back. Building and uploading AWS Lambda functions to the service is very simple and straight-forward, but now imagine you need to manage 100–150 web services to access a web page, multiplied by hundreds or thousands of web pages.

We believe that the only way to achieve this kind of flexibility and scale is automation and code reuse. These principles led us to build and open source DEEP Framework — a full-stack web framework that abstracts web services and web applications from specific cloud services — and DEEP CLI (aka deepify) — development tool-chain that abstracts package management and associated development operations.

Therefore, to make sure that the process of managing AWS Lambda functions is streamlined and automated, we have adopted a consistent approach to include 2 more files in each uploaded .zip:

'use strict';

import DeepFramework from 'deep-framework';
import Handler from './Handler';

export default DeepFramework.LambdaHandler(Handler);

{
  "name": "deep-todo-task-create",
  "version": "0.0.1",
  "description": "Create a new todo task",
  "scripts": {
    "postinstall": "npm run compile",
    "compile": "deepify compile-es6 `pwd`"
  },
  "dependencies": {
    "deep-framework": "^1.8.x"
  },
  "preferGlobal": false,
  "private": true,
  "analyze": true
}

Having these 3 files (Handler.es6, bootstrap.es6 and package.json) in each AWS Lambda function doesn't mean your final .zip file will be that small. Actually, a lot of additional operations happen before the .zip file is created. To name few:

• AWS Lambda performs better when the uploaded codebase is smaller. Since we provide both local development capabilities and one-step push to production, our process optimizes resources before deploying to AWS.
• ES6 is not supported by node.js v0.10.x that currently runs in AWS Lambda. We compile .es6 files into ES5 compliant .js files using babel.
• Dependencies that are defined in package.json are automatically pulled and fine tuned for node.js v0.10.x to provide best performance possible.

Putting Everything Together

First, you will need the following pre-requisites:

1. AWS Account (learn how to Create an Amazon Web Services Account)
2. AWS CLI (learn how to Configure AWS Command Line Interface)
3. Git v2+ (learn how to Get Started — Installing Git)
4. Java / JRE v6+ (learn how to JDK 8 and JRE 8 Installation Start Here)
5. Node.js v4+ (learn how to Install nvm and Use latest node v4)
6. DEEP CLI (execute in command line: npm install deepify -g)

Note: Don't use "sudo" in step 5. Otherwise you'll have to fix npm permissions.

Next, you will deploy the todo app using deepify:

1. deepify install github://MitocGroup/deep-microservices-todomvc ~/deep-microservices-todomvc
2. deepify server ~/deep-microservices-todomvc
3. deepify deploy ~/deep-microservices-todomvc

Note: When step 2 (deepify server) is finished, you can open in your browser http://localhost:8000 and enjoy the todo app running locally.

Cleaning Up

There are at least half a dozen services and several dozen of resources created during deepify deploy. If only there was a simple command that would clean up everything when we're done. We thought of that and created deepify undeploy to address this need. When you are done using todo app and want to remove web app related resources, simply execute step 4:

4. deepify undeploy ~/deep-microservices-todomvc

As you can see, we empower developers to build hassle-free cloud-native applications or platforms using microservices architecture and serverless computing. And what about security?

Security

Well, one of the biggest value propositions on AWS is out-of-the-box security and compliance. The beauty of cloud-native approach is that security comes by design (in other words, it won't work otherwise). We take full advantage of shared responsibility model and enforce security in every layer.

Developers and applications' end users benefit of AWS IAM best practices through streamlined implementations of least privilege access, delegated roles instead of credentials and integration with logging and monitoring services (e.g. AWS CloudTrail, Amazon CloudWatch, Amazon Elasticsearch + Kibana). For example, developers and end users of todo app didn't need to explicitly define any security roles (it was done by deepify deploy), but they can rest assured that only their instance of todo app will be using their infrastructure & platform & application resources.

Here below are 2 security roles (1 for back-end and 1 for front-end) that have been seamlessly generated and enforced in each layer:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["lambda:InvokeFunction"],
            "Resource": ["arn:aws:lambda:us-east-1:123456789000:function:DeepProdTodoCreate1234abcd*"]
        }
    ]
}

{
  "Version": "2015-10-07",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["deep.todo:task:create"],
      "Resource": ["deep.todo:task"]
    }
  ]
}

Benchmarking

We have been continuously benchmarking AWS Lambda for various use cases in our microapplications. After a couple of repetitive times doing similar analysis, we decided to build the benchmarking as another microapplication and reuse the ecosystem to automatically include where we needed it. The codebase is open sourced on Github:

Particularly, for todo app, we have performed various benchmarking analysis on AWS Lambda by tweaking different components in a specific function (e.g. function size, memory size, billable cost, etc.). Next, we would like to share results with you:

Performance

Speaking of performance, we find AWS Lambda mature enough to power large-scale web applications. The key is to build the functions as small as possible, focusing on a simple rule of one function to achieve only one task. Over time, these functions might grow in size, therefore we always keep an eye on them and refactor / split into the lowest possible logical denominator (smallest task).

Using the benchmarking tool, we ran multiple scenarios on the same function from todo app:

Based on performance data, we have learned pretty cool stuff:

• The smaller the function is, the better it performs; On the other hand, if more memory is allocated, the size of the function matters less and less
• Memory size is not directly proportional to billable costs; Developers can decide the memory size based on performance requirements combined with associated costs
• The key to better performance is continuous load, thanks to container reuse in AWS Lambda

Conclusion

In this article, we have presented a small web application that is built with AWS Lambda at core. Together we walked through the thinking process of identifying the front-end, the back-end and the data tiers required to build the todo app. We focused on the structure of AWS Lambda functions used in this app, as well as security, performance and benchmarking steps that we use to build enterprise-level web applications. You can fork the example code repository as a starting point for your own web applications.

If you have questions or suggestions, please contact us.