How to Manage and Automate Distributed Apps with AWS Lambda’s Step Functions
Lambda is one of the fastest-growing products AWS has released in recent years. At the last re:Invent, Lambda was a key focus for AWS, and one of the most exciting Lambda announcements was the release of Step Functions. In this post I discuss why you would want to use Step Functions, and how they can be used to manage complex interactions between services.
Lambda Enables Microservices
One of the key reasons to use AWS Lambda is to structure your application as a collection of microservices. From this perspective, your app is divided into separate components, or services, with each component handling a small portion of the overall logic. When loading your code into Lambda, each service is run by a Lambda function. Together, all the combined Lambda functions make up your application.
Source: AWS re:Invent
The reason you’d want to use the microservices architecture for your application is to manage each component individually. If each component is independent of other components, you can scale them faster, and more easily. You can define different levels of resource usage for each service, troubleshoot services more effectively, isolate issues, and build a more resilient app. And with Lambda, AWS handles all the infrastructure changes invisibly behind-the-scenes, so you can focus on architecting and delivering differentiating application services.
Challenges with Distributed Apps
While Lambda makes it easy to manage small applications comprised of small numbers of services, a complex distributed app may incorporate hundreds or even thousands of Lambda functions. The interdependencies between services needs to be carefully managed in a logical way, and this will vary for each application code path or inter-service interaction. Sometimes you may need the functions to execute in a linear way, one after another. At other times, you may want them to execute in parallel, all at one time. Either way, you’d have to define this at the individual function level, and this can be a daunting task because of the sheer number of functions you’ll have to manage. Enter Step Functions, a new service to simplify Lambda workflows.
Step Functions—Visual Workflow for Distributed Apps
The Step Functions service lets you visually or programmatically define the desired interactions and co-invocations between your Lambda processes. Below is an example of a Step Functions workflow.
Source: Amazon.com
With Step Functions, each component of your application is defined as a state machine. This means each component has a state which can be changed by input values. These states may be tasks, linear steps, parallel steps, branches, or timers. Together, each step defines how your app behaves. The Step Functions service gives you a visual console to control and manipulate these state machines easily, and they can also be provisioned and managed using CloudFormation.
Use Cases for Step Functions
Step Functions can be used to help build applications that contain complex business logic. Along with this, time-sensitive transactions that need to be automated can be easily handled by Step Functions. Whether it’s transaction processing for e-commerce websites, or intelligent monitoring systems, Step Functions makes it possible to build automated, yet highly complex applications.
For DevOps teams, automation is key to enabling the entire CI/CD process. With Step Functions, you can use events as triggers to automate changes in infrastructure, thus enabling DevOps at scale for complex software delivery pipelines with many moving parts.
Working with Step Functions
Step Functions simplifies workflow management by enabling blueprints for common workflows like ‘wait state’, ‘retry failure’ or ‘parallel.’ These blueprints enable you to make changes without writing a line of code for some steps.
State machines are defined using JSON text, and the output is also in JSON. Once your workflow has run, you can monitor it from the Step Functions console. Here, you can see the input and output for each state machine. The console colour-codes processes—green for success, and red for failure.
Source: Amazon.com
Much like other AWS services, Step Functions uses CloudWatch for monitoring and CloudTrail for logging. Pricing for step functions is separate from Lambda, and other AWS services. Pricing is per ‘state transition’—the step between one state machine and another in your workflow. AWS gives you 4,000 free state transitions per month, which is enough to get up and running.
Step Functions is built for large-scale applications. At re:Invent, Werner Vogels said that it currently handles billions of tasks for Amazon, so it’s been battle-tested. Starting with simple flows, you can create really complex processes and deliver outstanding user experiences with Step Functions. As you look to accelerate application updates, scale infrastructure automatically, and build resilient apps, Step Functions is worth considering. It may just be the easiest way to migrate to a microservices model.
