Why Should You Run All Your Tests in Docker?

Containers have become the de facto standard for deploying applications. Whether developers are deploying them in the public cloud or a data center, containerization can bring a lot of benefits like a consistent environment, repeatable builds, and easy deployment patterns.

For those who want to learn about Docker and container fundamentals to deploying entire platforms to the cloud using continuous integration and continuous delivery (CI/CD), we recommend Scott Surovich and Marc Boorshtein’s book Kubernetes and Docker - An Enterprise Guide.

But so far, the focus has been on running applications in containers, with very little attention given to running tests in containers.

In this article, we’ll see how you can leverage containers for tests as well and why this could prove beneficial to your organization. We’ll also discuss how to effectively handle CI test observability and CI test debugging needed for containerized tests.

Pros of Running Tests in Containers

Running test cases in containers offers a brilliant value proposition with some great advantages:

• Consistent environment: With identical containers in every environment, there is no drift in software versions across dev and test environments. This significantly increases developer velocity and helps deliver code faster.
• Isolation: Containers provide an ephemeral environment to run your tests, making sure your tests run in isolation and don't interfere with one another.
• Repeatable builds: Containers are built against a consistent definition file, which can be used to recreate containers anytime.
• Quality CI pipelines: Having containerized test cases makes it easy to set up CI pipelines and maintain high-quality builds.

Demo Setup

Before we jump into the demo, you’ll need to make sure you have some prerequisites set up:

• Java 8+ for a sample Java application and unit tests
• Maven 3.6.0+ to download your dependencies and run your tests
• Docker to create a Docker image to run your tests

Code Setup

This demo will entail a simple Java application with unit tests written in jUnit5.

First, create the project using Maven:

mvn archetype:generate -DgroupId=com.example 
<-DartifactId=test-in-containers>
<-DarchetypeArtifactId=maven-archetype-quickstart>
<-DinteractiveMode=false>

This will create the folder structure with a sample Java project.

Delete the already existing code files at src/main/java/App.java and src/test/java/AppTest.java.

Next, replace the pom.xml with the following content:

<?xml version="1.0" encoding="UTF-8"?> 
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion> 
        <groupId>org.example</groupId>
    <artifactId>test-in-containers</artifactId>
        <version>1.0-SNAPSHOT</version>
        <dependencies>
            <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>RELEASE</version>
                <scope>test</scope>
            </dependency>
        </dependencies>
 
        <build>
            <plugins>
                <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                    <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    </configuration>
                </plugin>
 
                <plugin>
                      <groupId>org.apache.maven.plugins</groupId>
                  <artifactId>maven-surefire-plugin</artifactId>
                  <version>3.0.0-M5</version>
                  </plugin>
                  </plugin>
            </plugins>
        </build>
</project>

This configures Maven to use Java 1.8 and add jUnit5 as a dependency and also adds the Maven Surefire plugin for running tests.

Now, create the source code file at src/main/java/Calculator.java:

public class Calculator {
 
        public int add(int a , int b){
            return a + b;
        }
 
}

And create the test file at src/test/java/CalculatorTest.java:

import org.junit.jupiter.api.Test;
 
import static org.junit.jupiter.api.Assertions.*;
 
class CalculatorTest {
 
        @Test
        void AddTwoNumbers() {
            Calculator calc = new Calculator();
            int result = calc.add(1,1);
            assertEquals(2, result);
        }
 
        @Test
        void AddNegativeNumbers() {
            Calculator calc = new Calculator();
            int result = calc.add(-1,-1);
            assertEquals(-2, result);
        }
 
        @Test
        void AddZeroToANumber(){
            Calculator calc = new Calculator();
            int result = calc.add(1,0);
            assertEquals(1, result);
        }
}

And that’s it. Your system and code are ready to proceed to containerization.

Containerize the Tests

In this section, you’ll see how to containerize your tests, after which DevOps can run them in your CI pipeline.

First, create a Dockerfile, which you will use to create a Docker image to be run in your CI pipelines:

FROM maven:3.8.1-openjdk-11-slim
WORKDIR /src
ENTRYPOINT ["mvn", "clean"]

Notice, you’re using Java 11 and Maven 3.8.1 to build the Docker image. You can create many such images with different versions of Java and Maven, which can then run in a single test environment, reducing the maintenance cost of different environments and eliminating interference due to different versions.

Build the docker image:

docker build -t test-in-containers:1.0-3.8.1-openjdk-11-slim .

Here, you are adding Maven and Java versions in the Docker image file tag for easy referencing.

Once the image is built, you can run the tests via the following command:

docker run -v `pwd`/test-in-containers:/src test-in-containers: 1.0-3.8.1-openjdk-11-slim test

Notice how you are mounting the source code inside the container using Docker volumes. You can then hand over the image to DevOps to set up these tests to run on every commit.

You can also mount another volume and generate test reports, but for the purpose of this demo, you will stick to just running tests.

In the prerequisites section, it was noted that you need a certain version of Java and Maven to make sure your tests run properly. Over time, you might need different versions for different projects, and it can get challenging for DevOps to maintain consistent test environments.

For example, one of your applications might need Java 8 and the other might need Java 11. Having them both installed on the same machine might cause interference, or if you run tests on a single Java version, you run the risk of shipping buggy code.

Another scenario might be that you want to test your code for backward or forward compatibility with separate Java versions; for that, DevOps will also need to carefully maintain different environments.

In all of these scenarios, containerizing your unit tests can help alleviate the problem of maintaining different test environments. DevOps just needs to make sure that they have an environment where Docker containers can run and the rest is managed by the developers themselves. Tests running in containers can solve this age-old problem of dev-test environment drift.

Note: We’ve covered how to containerize unit tests, but the same principles can be applied to other tests, namely integration and end-to-end.

Visibility in a CI Pipeline

Running tests in containers is not all rosy and come with its own set of challenges. One of the issues that developers and DevOps face while running their tests in a CI pipeline is test observability, including CI test troubleshooting and debugging. You might have test failures you want to be notified about and resolve, or you may experience a slower build that’s hampering deployment frequency. But there is no straightforward way to get a peek into what is happening inside your CI pipeline.

That’s where Foresight comes into play. Foresight is a developer tool that can help you gain CI observability. It provides CI monitoring and troubleshooting capabilities, which can help you lower your build times and increase deployment frequency. Foresight provides a dashboard where you can monitor your CI workflows, identify slow and unreliable tests, assess the level of risk of software changes, and reduce testing cycle time dramatically.

Foresight also keeps track of all your test runs and KPIs associated with those runs—things like how much time each test has taken and the reason for slow tests or test failures. Foresight automatically detects the changed part of the code which is untested and assesses the level of risk that can be introduced. Furthermore, it offers a prioritization of the tests to optimize the build times.

Conclusion

Running your tests (unit, integration, or E2E) in containers alleviates the pain of maintaining cumbersome test environments and helps you streamline your CI pipelines. Containerizing tests can save your organization hours of having to debug issues caused by mismatched software versions. And although CI observability and test debugging of tests can be an issue, Foresight helps you gain rapid and detailed insights into key metrics for your CI pipelines to keep them on track.

Lastly; if you have yet to take your first step into the world of Foresight, see the pricing and you can begin your journey here.