How to Locate Slowdowns in Your CI Pipeline

Development velocity is vital in today’s SaaS world. The faster you can iterate, the faster you can find killer new features for your service.

But this need for speed can work against the imperative to maintain high-quality deliverables. Testing is one way to maintain quality but is often seen as a barrier to new releases. End-to-end tests are especially prone to be slow and hard to debug. If your project has accumulated such tests, you’ve probably noticed your development velocity go down the drain.

So, how can you stay on top of developing new features while maintaining a quality product? Monitoring your CI pipeline will give you important notifications when things slow down. Going a step further, a monitoring tool optimized for CI pipelines can help you locate the culprit of the slowdown and fix it faster than a generic monitoring solution would.

In this article, we’ll demonstrate how to monitor a GitHub Actions CI pipeline with Foresight, locate slowdowns, and fix them right away.

Prerequisites

This tutorial will use a pre-built AWS CDK project as a base. It requires a GitHub and AWS account, and you’ll need a Node.js installation with npm to run the commands.

The tutorial also assumes that you’re running the examples in Cloud9, a pre-configured cloud IDE by AWS that comes preloaded with all of the required software.

Forking the Repository to Your GitHub Account

You can access our example repository, which has a CDK project, end-to-end tests, and pre-configured GitHub actions. You’ll have to fork this repository into your GitHub account to modify it as well as the actions that run with your credentials.

The repository includes a CDK stack with the following resources:

• VPC resources for the RDS instance
• An RDS instance with PostgreSQL
• A Lambda function with a simple output
• A Lambda function that accesses the RDS instance
• An API Gateway to make everything accessible from the outside

Clone the Repository Locally

The example repository includes one GitHub action for running the tests in CI, but the deployment happens from your local machine. This requires you to clone the repository from your GitHub account, which you can do with the following command:

$ git clone <YOUR_FORKED_REPO_URL>

After running this command, you can start browsing the code in your preferred editor.

Install Dependencies

Next, you’ll need to use the command below to install the dependencies so that the deployment can run without interruption. These include packages like TypeScript, Jest, CDK, and others.

$ npm i

After this command finishes executing, you are ready to deploy this stack to AWS.

Setting up Foresight

Before deploying the infrastructure, you’ll need to set up Foresight.

Let's signup for Foresight and connect our pipeline to it. First, we need to install Foresight's GitHub app into our account. Then we need to pick the repositories we want to monitor on Foresight. After selecting which repositories we want to monitor, we then have to create our first project on Foresight.

To monitor our tests on Foresight, we need another small step to complete, which is uploading our test reports to Foresight.

Because Foresight helps us to upload our test run reports automatically to monitor our test analytics, we just need to configure each of the GitHub Actions that we choose to watch. We can use the same configuration for all of them. Foresight automatically merges our workflow and test run data. We pick our test framework and update our YAML with making sure that our tests run before the test report uploader step.

That's it we're all done!

Deploying the Infrastructure

The infrastructure is deployed with the CDK using the command below.

Be aware that this deployment can take over 10 minutes.

$ npm run deploy

After running this command, the application is accessible via HTTP on the URL in the outputs.json.

Running the Tests Locally

To check that everything worked, you can run the tests locally with the following command:

$ npm run test

There are four tests defined, two for every endpoint. The first one executes the cold-start, and the second one runs twenty requests after the Lambda function is warmed up. The index endpoint is handled by a Lambda function that only gives a text output while the database endpoint connects to the RDS instance to read a timestamp.

Pushing the Outputs

We’ll do that with the following command:

$ git add -A
$ git commit -m "Add stack outputs."
$ git push

At this point, the test run will start automatically, and Thundra Foresight will pick up the results. This run takes a few minutes to complete.

Finding Slow Tests with Foresight

Now, when you look at the Foresight web console, you should find your first test run inside the RDS Example project. If you click on the test run, you’ll be presented with a list of slow tests sorted by the runtime, like the one shown in the figure below.

If we want to optimize our tests, we can go through this list from top to bottom.

Optimizing the Slowest Test

If we click on the slowest test in our list, we’ll see the time that every run of that test took. This simple example isn’t particularly interesting because it just tells us the test took around four seconds.

Foresight reveals that every request took more than 130 milliseconds. Most of this time was spent inside the database Lambda function, and only a fraction of it was an actual request to our database. This implies that the performance problem lies inside our Lambda function code.

We’ve prepared an optimized implementation that only executes init tasks, like loading the database credentials and connecting to the database, once.

const aws = require("aws-sdk");
const postgres = require("pg");

let db;
async function initDatabaseConnection() {
  console.log("Initializing database connection!");
  const secretsManager = new aws.SecretsManager({
    region: "eu-west-1"
  });

  const dbSecret = await secretsManager
    .getSecretValue({ SecretId: process.env.DB_SECRET_ARN })
    .promise();

  const dbCredentials = JSON.parse(dbSecret.SecretString);

  db = new postgres.Client({
    host: dbCredentials.host,
    port: dbCredentials.port,
    user: dbCredentials.username,
    password: dbCredentials.password,
  });

  await db.connect();
}

exports.handler = async (event) => {
  if (!db) await initDatabaseConnection();

  const result = await db.query("SELECT NOW()");

  console.log(result);
  
  return {
    statusCode: 200,
    body: JSON.stringify(result),
  };
};

After you have updated the file, rerun the deployment with the following command:

$ npm run deploy

If you commit the changes and push them again, the test will be rerun with GitHub Actions.

$ git add -A
$ git commit -m "Improve performance"
$ git push

Now, Foresight shows how our time inside the Lambda function went down to just 18 milliseconds. Because Lambda is billed by millisecond, this is better for us and our clients.

Conclusion

Development velocity is an important component of innovation, but it shouldn’t come at the expense of code quality. That’s why monitoring your delivery pipeline is essential—and using a monitoring system that has been tailored for test CI monitoring is even better. With such a specialized tool, you can quickly find and fix delivery problems for better products at a lower cost.

Foresight is just such a tool. It works with the popular JavaScript testing framework Jest as well as GitHub Actions. It has aa applications served on GitHub Marketplace. Gain visibility into your CI pipelines with Foresight so you can find and fix slowdowns with ease.