How to Locate Slowdowns in Your CI Pipeline
Development velocity is vital in today’s SaaS world. The faster you can iterate, the faster you can find killer new features for your service.
But this need for speed can work against the imperative to maintain high-quality deliverables. Testing is one way to maintain quality but is often seen as a barrier to new releases. End-to-end tests are especially prone to be slow and hard to debug. If your project has accumulated such tests, you’ve probably noticed your development velocity go down the drain.
So, how can you stay on top of developing new features while maintaining a quality product? Monitoring your CI pipeline will give you important notifications when things slow down. Going a step further, a monitoring tool optimized for CI pipelines can help you locate the culprit of the slowdown and fix it faster than a generic monitoring solution would.
In this article, we’ll demonstrate how to monitor a GitHub Actions CI pipeline with Foresight, locate slowdowns, and fix them right away.
This tutorial will use a pre-built AWS CDK project as a base. It requires a GitHub and AWS account, and you’ll need a Node.js installation with npm to run the commands.
The tutorial also assumes that you’re running the examples in Cloud9, a pre-configured cloud IDE by AWS that comes preloaded with all of the required software.
Forking the Repository to Your GitHub Account
You can access our example repository, which has a CDK project, end-to-end tests, and pre-configured GitHub actions. You’ll have to fork this repository into your GitHub account to modify it as well as the actions that run with your credentials.
The repository includes a CDK stack with the following resources:
- VPC resources for the RDS instance
- An RDS instance with PostgreSQL
- A Lambda function with a simple output
- A Lambda function that accesses the RDS instance
- An API Gateway to make everything accessible from the outside
Clone the Repository Locally
The example repository includes one GitHub action for running the tests in CI, but the deployment happens from your local machine. This requires you to clone the repository from your GitHub account, which you can do with the following command:
After running this command, you can start browsing the code in your preferred editor.
Next, you’ll need to use the command below to install the dependencies so that the deployment can run without interruption. These include packages like TypeScript, Jest, CDK, and others.
After this command finishes executing, you are ready to deploy this stack to AWS.
Setting up Foresight
Before deploying the infrastructure, you’ll need to set up Foresight.
Let's signup for Foresight and connect our pipeline to it. First, we need to install Foresight's GitHub app into our account. Then we need to pick the repositories we want to monitor on Foresight. After selecting which repositories we want to monitor, we then have to create our first project on Foresight.
To monitor our tests on Foresight, we need another small step to complete, which is uploading our test reports to Foresight.
Because Foresight helps us to upload our test run reports automatically to monitor our test analytics, we just need to configure each of the GitHub Actions that we choose to watch. We can use the same configuration for all of them. Foresight automatically merges our workflow and test run data. We pick our test framework and update our YAML with making sure that our tests run before the test report uploader step.
That's it we're all done!
Deploying the Infrastructure
The infrastructure is deployed with the CDK using the command below.
Be aware that this deployment can take over 10 minutes.
After running this command, the application is accessible via HTTP on the URL in the outputs.json.
Running the Tests Locally
To check that everything worked, you can run the tests locally with the following command:
There are four tests defined, two for every endpoint. The first one executes the cold-start, and the second one runs twenty requests after the Lambda function is warmed up. The index endpoint is handled by a Lambda function that only gives a text output while the database endpoint connects to the RDS instance to read a timestamp.
Pushing the Outputs
We’ll do that with the following command:
At this point, the test run will start automatically, and Thundra Foresight will pick up the results. This run takes a few minutes to complete.
Finding Slow Tests with Foresight
Now, when you look at the Foresight web console, you should find your first test run inside the RDS Example project. If you click on the test run, you’ll be presented with a list of slow tests sorted by the runtime, like the one shown in the figure below.
If we want to optimize our tests, we can go through this list from top to bottom.
Optimizing the Slowest Test
If we click on the slowest test in our list, we’ll see the time that every run of that test took. This simple example isn’t particularly interesting because it just tells us the test took around four seconds.
Foresight reveals that every request took more than 130 milliseconds. Most of this time was spent inside the database Lambda function, and only a fraction of it was an actual request to our database. This implies that the performance problem lies inside our Lambda function code.
We’ve prepared an optimized implementation that only executes init tasks, like loading the database credentials and connecting to the database, once.
After you have updated the file, rerun the deployment with the following command:
If you commit the changes and push them again, the test will be rerun with GitHub Actions.
Now, Foresight shows how our time inside the Lambda function went down to just 18 milliseconds. Because Lambda is billed by millisecond, this is better for us and our clients.
Development velocity is an important component of innovation, but it shouldn’t come at the expense of code quality. That’s why monitoring your delivery pipeline is essential—and using a monitoring system that has been tailored for test CI monitoring is even better. With such a specialized tool, you can quickly find and fix delivery problems for better products at a lower cost.