Creating Scalable Campaigns Using AWS S3, Amplify, SQS, DynamoDB, and Event Bridge

Introduction

One of our clients does marketing campaigns for some pretty big clients. Often these campaigns will have hundreds of thousands of submissions in a very short time. Many of these involve customers submitting pictures of receipts or products in order to be eligible for some sort of giveaway or discount code. Problems arose with the high volume of submissions being processed through a typical website form that saved data directly to a database, and an image to local storage on the server. Hundreds of thousands of images being stored on your server will fill up space very quickly! Not to mention the processing involved for each submission, many going on simultaneously. It’s safe to say that during peak times the server would be very slow and customers would complain back to the client that it was sometimes almost unusable. It was up to us to come up with a solution. Below is an explanation of how we were able to help this customer scale their campaigns and improve performance.

The Tools

The tools we use are important here because they allow us to scale quickly and efficiently! AWS is the platform we chose for this because of familiarity, and the customer already had an AWS account which made it a no-brainer. You can most likely find similar products to these at competing cloud platforms such as Google Cloud and Microsoft Azure.

AWS API Gateway - AWS API Gateway is a fully managed service that makes it easy for developers to create, publish, secure, and monitor APIs at any scale.

AWS Amplify - AWS Amplify is a comprehensive development platform from Amazon Web Services that enables developers to build, deploy, and manage scalable mobile and web applications quickly and easily. It offers a suite of tools and services, including a fully managed backend, hosting, and CI/CD pipelines, designed to streamline development workflows. We used Amplify to host the front-end components which was the landing page form and client-side code. Amplify also uses other AWS services on the backend. Using the amplify console you can add/configure a domain (Route 53) for your Amplify app, as well as caching (Cloudfront), SSL (Certificate Manager), and a CI/CD pipeline (Code Deploy).

Blazor (Microsoft) - Microsoft Blazor is a web framework that enables developers to build interactive, client-side web applications using C# instead of JavaScript. Part of the ASP.NET family, Blazor allows developers to create single-page applications (SPAs) that run in the browser or on the server with a shared codebase. Blazor give you all the advantages of using dotnet and JavaScript. We chose this because we have some keys and data that we don’t want to appear in the browser by snooping through JavaScript files. Using Visual Studio and it’s GitHub integration we can perform updates, sync to the repository, and then push to the remote where it will trigger Amplify to build and deploy a new version. This saved a ton of development time for small changes.

AWS SQS - Amazon Simple Queue Service (SQS) is a fully managed message queuing service by AWS that enables applications to communicate and coordinate by exchanging messages asynchronously. We used SQS to store the submission data from the user. When a user would submit, rather than have them wait while we do all of the processing, we simply fire it off to the queue and it gets put in line for processing. This means that the customer almost always has near instantaneous feedback for their submission, and the heavy processing is off loaded t

AWS DynamoDB - Amazon DynamoDB is a fully managed, serverless NoSQL database service provided by AWS, designed for high-performance applications that require low-latency access to large amounts of data.

AWS S3 - Amazon Simple Storage Service (S3) is a scalable, secure, and durable object storage service provided by AWS, designed to store and retrieve any amount of data from anywhere on the web.

AWS EventBridge - AWS EventBridge is a serverless event bus service that enables applications, microservices, and AWS services to communicate with each other using events.

The Process

This is not an article that will provide code examples, we’re not here to teach you how to code, but rather give you an idea of an architecture that will help you scale for a large campaign. That said here is a 10,000 foot view of what we are working with.

API Gateway

First we start with the API Gateway. All of our server side stuff is being done on an API Gateway which is attached to a Lambda Function. Visual Studio makes it very easy to create an API project and using the AWS Toolkit deploy that API to Lambda/API Gateway. In here we do all of our call outs to the database and to other AWS Services such as S3 for File storage, DynamoDB to store our entries, SQS to create our processing queues, and SES to send out the confirmation emails to the users. You will need you gateway to do a couple of different things. If you require a file upload, you will need to have an api endpoint to process that file upload and return an S3 URL. You will also need an endpoint to process the rest of the submission. Most often this submission will receive a JSON payload consisting of users attributes such as first name, last name, email, zip code, phone number, uploaded image URL etc.

Client Side Setup

Next we need to setup up our form. As I mentioned before we are using Visual Studio’s Blazor framework for building one page applications. We are using this because we like c# and it can store things like API keys without exposing them in typical javascript. This is important as we will want to setup our API gateway to authenticate via API key to make sure your endpoints are only getting data from authenticated sources. If your submission form is requiring an image such as a receipt to be uploaded we often build that out first. Your API gateway will need to interact with S3 for file storage (or use your cloud storage platform of choice). There are many solutions for providing uploads via JavaScript from the client side. Upon upload it will go to your API, upload the image (you should put some sort of validation there to make sure people are uploading images) and return a URL (in our case an S3 URL). Once we get that back we store that variable in a hidden input field while the user fills out the rest of the form. When a user fills out the rest of the form, they click submit and the data for that submission hit’s your API endpoint for processing the submission. Depending on your requirements, you’ll probably do some validation here too in order to keep your data consistent. For hosting and deployment we are using AWS Amplify which will link to your github repo for your project and automatically perform the CI/CD operations for deploying changes. Simply update your local version, push your changes to the repo, then sync the changes to your remote repo and Amplify will do the rest. The initial setup of Amplify can take a bit of work, but once it’s working it couldn’t be easier to deploy changes.

Queueing and Processing

From here the user’s data has now left your form, and has been processed through your API gateway and entered in to an SQS queue. SQS queues are great because they allow us to let entries accumulate without putting extra pressure on your API gateway to process. The SQS queue item itself is a JSON respresentation of the user’s entry.

EventBridge the Real MVP

In order to process our SQS queue items, we have used Event Bridge Pipes to link to your SQS queue. In our case we also created an API endpoint within the event bridge “integrations” area for that pipe to process the entries. That API endpoint is as you probably have guessed, in the API gateway that we setup at the beginning. This endpoint in the API gateway sends the data to our Dynamo DB table we have setup for the submissions. So the flow looks something like this.

User Form -> API Gateway -> SQS -> EventBridge Pipe -> API Gateway -> DynamoDB

This works very well for very large campaigns. We processed over 250K submissions in less than a week without even a single hiccup using this method. Depending on scale you may need to play around with the settings in your API Gateway. By and large the SQS queue and Event Bridge will do the bulk of the work and allow your customers to get a very quick response when submitting their entries to your campaign.

If you’re interested in finding out more, or need consulting on scaling your campaigns without using expensive SAAS options then shoot us an email.