Run Synthetic tests in your CI/CD pipelines with the Datadog CircleCI orb

CircleCI is a CI/CD service that allows organizations to rapidly build, test, and deploy within their pipelines on a single platform. If you are using CircleCI for your CI/CD pipelines, you can now leverage the Datadog Synthetics CircleCI orb to implement Synthetic tests as part of shift-left testing. CI/CD testing is a widely adopted DevOps standard that helps teams mitigate any potential issues that could arise as a result of faulty code deployments.

Datadog Synthetic Monitoring enables you to implement a suite of end-to-end tests in your CI/CD pipelines, so you can automatically detect and block the deployment of code changes that would break key workflows or endpoints. The Datadog CircleCI orb enables you to implement these Synthetic tests in your CircleCI pipelines, surfacing issues that you can then diagnose and debug with Datadog. Adding Synthetic tests to your CircleCI pipelines will help you ensure that new code deployments are reliable, preempting any potential problems for your end users.

In this post, we’ll show you how to set up the CircleCI orb to run Synthetic tests, debug failing Synthetic tests, and monitor CircleCI pipelines together with Datadog’s existing CircleCI integration.

Set up the Datadog CircleCI orb to run Synthetic tests

A CircleCI orb is an open source, reusable config file for automating processes, expediting project startup, and easing integration between your CircleCI pipelines and services across the stack. To get started, simply add your Datadog API and application keys as environment variables to your CircleCI project. You can then start configuring the Datadog CircleCI orb to execute Datadog Synthetic tests in your pipelines.

To make sure that code deployments don’t introduce unexpected issues for your end users, you can implement a suite of Synthetic end-to-end tests to confirm that your application is functioning as expected. For example, the CircleCI pipeline configuration below utilizes the Synthetics CircleCI orb.

version: 2.1

orbs:
  synthetics-ci: datadog/synthetics-ci-orb@1.0.1

jobs:
  e2e-tests:
    docker: 
      - image: cimg/base:stable
    steps:
      - synthetics-ci/run-tests:
          public_ids: 'abc-d3f-ghi, jkl-mn0-pqr'

workflows:
  run-tests:
    jobs:
      - e2e-tests

The above configuration will automatically trigger the Synthetic tests associated with the specified public_ids every time you deploy code changes. You can configure the orb to block the deployment if the tests fail using a config override or through the Synthetics UI. The results of your Synthetic tests will appear in your CircleCI pipeline (as shown in the example below).

If you want to run tests on applications that are not publicly available (e.g., internal applications), you can also configure local testing by starting a local server and triggering Synthetic tests using the datadog-ci tunnel (as shown in the example below).

version: 2.1

orbs:
  synthetics-ci: datadog/synthetics-ci-orb@1.0.1

jobs:
  e2e-tests:
    docker:
      - image: your-image
    steps:
      - checkout
      - run:
          name: Running server in background
          command: npm start
          background: true
      - synthetics-ci/run-tests:
          config_path: tests/tunnel-config.json
          files: tests/*.synthetics.json
          test_search_query: 'tag:e2e-tests'
          tunnel: true

workflows:
  test-server:
    jobs:
      - build-image
      - integration-tests:
          requires:
            - build-image

By virtue of the orb’s reusability, you can leverage the same unique test suite in development, staging, and production to simplify and standardize your testing across different environments. For example, you can configure your tests to automatically run in your dev environment after they’ve passed locally to ensure that new features work properly before deployment. Simply override the startURL in your global config file so that it points to your development environment.

Debug failing Synthetic tests

Datadog Synthetic Monitoring integrates with APM to offer invaluable context for troubleshooting failing tests (e.g., duration, device, and browser) alongside metrics, logs, and other data. For example, you can inspect a test’s correlated trace to learn more about why it failed. In this case, the trace shows you that the request to your API returned a 404 error. Upon further investigation, you find that this request was being transmitted to a recently deprecated client library, so you may be able to resolve this issue by switching to a different provider.

Improve the reliability and efficiency of your CircleCI pipelines with Datadog

Using the Datadog CircleCI orb in conjunction with Datadog’s CircleCI integration allows you to monitor your CircleCI pipeline performance. In addition to running Synthetic tests in CircleCI, you can also use Datadog CI Visibility to optimize your pipelines. Once you’ve instrumented your CircleCI pipelines with CI Visibility, health and performance metrics will automatically stream into Datadog. You can view the results of your jobs in the CI Results Explorer and investigate problematic jobs to see where errors and bottlenecks might be occurring.

Each job contains a batch of test IDs specified by the orb. Once Datadog has run the tests, you can click on a batch of results—tagged with useful metadata, including status, branch, and duration—to pinpoint the specific test(s) that failed. As in the image below, Datadog itemizes the test results from the orb’s execution, allowing you to compare the results of individual test runs across different browsers, devices, and locations.

CI Visibility shows you which pipelines are the slowest or most error-prone so you can take action to improve the reliability of your tests and builds. Additionally, CI Visibility automatically detects flaky tests—another common source of failing jobs. As in the example below, you can inspect the erroneous span to get more helpful context for troubleshooting the failure.

Test with the CircleCI orb and monitor your pipelines with Datadog

The Datadog CircleCI orb enables you to easily incorporate Synthetic tests into your CircleCI pipelines, allowing all of your development teams to quickly detect issues before they degrade your user experience. This integration extends our existing support for Synthetic Monitoring in your CI pipelines, complementing the Datadog GitHub Action and Datadog plugin for Jenkins. CI Visibility also provides rich context around the health of your CircleCI pipelines, enabling you to optimize your CI/CD workflows. If you’re new to Datadog, sign up for a 14-day free trial.

-a