Modern continuous integration (CI) practices enable development teams to quickly and efficiently build and deploy application code to a shared codebase. However, deploying new code is typically accompanied by tests, and as the codebase expands, this results in a proportionately larger test suite. When an entire test suite needs to run in order to validate each small code change, testing can feel more like a chore rather than an integral part of the development process, creating a cumbersome feedback cycle for developers.
Datadog Testing Visibility gives you deep insights into each commit and automatically detects flaky tests that are compromising your build. Now, we’re excited to announce Datadog Test Impact Analysis (formerly known as Intelligent Test Runner), a new feature for CI Visibility that helps you accelerate the testing process by identifying and running only the relevant tests for each commit. By enabling Test Impact Analysis for your test services, you can reduce both testing time and resources required, and minimize the risk of broken pipelines caused by irrelevant flaky tests.
In this post we’ll cover how to:
- Create faster testing feedback cycles with intelligent test selection
- Visualize resource savings with our out-of-the-box dashboard
Create faster feedback cycles with intelligent test selection
Large codebases typically require a proportionately large test suite, which equates to lengthy testing times for each code change regardless of how big or small the change is. This can create a slow feedback cycle for developers, who spend valuable development time waiting since they usually need to monitor testing until the pipeline is complete.
Datadog Test Impact Analysis automatically selects and runs only the relevant tests needed to validate any given commit, enabling you to reduce the amount of time spent testing while maintaining test coverage. Test Impact Analysis analyzes your test suite to determine the code each test covers, and then cross-references that coverage with the files impacted by a new code change. Datadog uses this information to run a selection of relevant, impacted tests, omitting the ones unaffected by the code change and reducing the overall testing duration.
By simply selecting the test services you’d like to enable Test Impact Analysis for in the CI Test Service Settings, Datadog will apply intelligent test selection to that service’s commits. You can also choose which branches you’d like to exclude from Test Impact Analysis within a test service. Running every test on your default branch is still recommended as a failsafe practice—as a part of this recommendation, your default branch is preset to be excluded from Test Impact Analysis, however, you can still configure it to be included.
Minimize disruptions due to flaky tests
By minimizing the number of tests run per commit, Datadog Test Impact Analysis can help reduce the frequency of flaky tests disrupting your pipelines. Flaky tests are tests that may pass or fail at random given the same commit. This can be particularly frustrating when the test flaking is unrelated to the code change being tested. After enabling Test Impact Analysis for your test services, you can limit each commit to its relevant tests to ensure that flaky tests unrelated to your code change don’t end up arbitrarily breaking your build. By monitoring your commits, Datadog CI Visibility will automatically identify any flaky tests affecting your CI/CD workflow.
Get visibility into resource savings and slow tests
Datadog measures your Test Impact Analysis performance against baseline testing times to calculate the time saved from unnecessary testing. This enables developers to more frequently integrate new application code by reducing the time spent testing. Once you enable Test Impact Analysis for your test services, you can begin visualizing these resource savings in our out-of-the-box dashboard. The dashboard breaks down the time saved across your services, repositories, commits, and authors so you can better understand where Test Impact Analysis is the most effective.
You can also view your time saved with Test Impact Analysis against your testing time within the Tests View page or within a test session’s details.
Since branches with Test Impact Analysis enabled will only run relevant tests, it can greatly reduce the total number of tests run per commit. By drilling down into a specific commit with Test Impact Analysis enabled, you gain a concentrated view into the tests impacted by your code change, enabling you to more quickly surface a relevant failed or slow test affecting your build.
Start accelerating your software delivery process today
Using Datadog Test Impact Analysis, you can begin to create faster developer feedback cycles and reduce the risk of your pipelines failing due to unrelated flaking tests. Since Test Impact Analysis is built into Datadog CI Visibility, you can reap its time-saving benefits while monitoring your pipelines and tests from the same platform. This feature is now generally available to all Datadog customers—you can learn more in our documentation. For more information about troubleshooting your pipelines and tests with CI Visibility, check out our blog post.
If you aren’t already a Datadog customer, you can sign for up for free 14-day trial to begin monitoring your application today.