Security teams today normally perform ongoing vulnerability remediation as a key part of their efforts to secure applications. This process entails applying updates to remove known flaws, typically published as Common Vulnerabilities and Exposures (CVEs), that are discovered in third-party libraries within application code. While “applying updates” might sound like a straightforward task, in practice, eliminating vulnerabilities has become increasingly challenging in the current environment. Two of the obvious factors complicating vulnerability remediation are the rising number of vulnerabilities and the accelerating rate of software updates throughout the application lifecycle.
However, it is a third factor—the growing complexity of modern architecture—that most urgently calls for a new approach to vulnerability remediation. The conventional strategy that most security teams embrace today amounts to addressing vulnerabilities one by one in isolation, but that game plan is simply no longer viable. Modern applications are usually built from libraries that in turn depend on other libraries, and so on, and these dependency trees are constantly evolving over time as new updates are introduced. Under these conditions involving complex dependencies, when you address one vulnerability in isolation, you can easily introduce another that is even more critical than the one you’re trying to fix.
To help customers solve this problem, the Datadog platform offers smart vulnerability remediation, an approach and set of capabilities available today in our Software Composition Analysis (SCA) and Cloud Security Management (CSM) products. Vulnerability remediation in Datadog is designed to assess vulnerabilities collectively and optimize the actions users should take to address them.
In this blog post, we will discuss:
- The risks of the traditional approach to vulnerability remediation
- How a smarter strategy would require considering an application’s entire dependency tree
- How Google’s OSV Guided Remediation feature offers an improvement to the traditional approach—but with some important limitations
- How Datadog’s smarter approach to vulnerability remediation significantly improves upon the traditional approach and OSV Guided Remediation
Addressing vulnerabilities individually: a risky practice
Traditional vulnerability remediation involves removing each vulnerability one at a time—a shortsighted approach that is often counterproductive. To take a simple example, imagine a scenario where your application is using Apache Log4j version 1.2.17, which you learn is affected by vulnerability CVE-2020-9488. According to its official description, this is a medium-level severity rating that is fixed in Log4j 2.12.3 and 2.13.1. If you focus on this one vulnerability in isolation, it might seem perfectly reasonable to upgrade to Log4j version 2.13.1 to resolve this problem. However, this upgrade would introduce a new, severe risk: the infamous CVE-2021-44228 or Log4shell vulnerability, which carries a critical severity score of 10.
Here at Datadog, we are fortunate to have access to data that allows us to gauge the effectiveness of performing this type of piecemeal vulnerability remediation. Taking into account all third-party library updates, we have calculated the following:
- 36 percent of updates lead to versions that are still vulnerable.
- 11 percent of updates introduce new vulnerabilities that were not present before.
- Two percent of updates introduce a vulnerability whose severity score is even higher than the original being removed.
- Updating a library without considering its context results in a 30–40 percent increase in the number of updates needed to secure the application.
To improve upon this traditional strategy, teams need to take a broader view of dependencies and assess which available update(s) will best reduce an application’s overall vulnerability profile.
Towards a smarter approach: understanding direct and indirect dependencies
When building applications, software developers frequently rely directly on third-party libraries, which can be either open source or commercial components. These libraries represent direct dependencies. In Java applications, for example, you can see these direct dependencies with the help of Maven. An example set of these direct dependencies is shown below:
pom.xml
<dependency>
<groupId>net.jodah</groupId>
<artifactId>failsafe</artifactId>
<version>2.4.4</version>
</dependency>
<dependency>
<groupId>org.threeten</groupId>
<artifactId>threetenbp</artifactId>
<version>1.5.1</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.13.3</version>
</dependency>In addition, these direct dependencies often rely on other third-party libraries, termed transitive or indirect dependencies. These dependencies are libraries, packaged with your applications, that your services depend on in an indirect way. For example, a service might depend directly on a first library, and this first library might in turn directly depend on a second library. The second library would thus be an indirect or transitive dependency of the original service, as illustrated below:
Why look beyond direct dependencies?
While the traditional approach to vulnerability remediation looks only at direct dependencies, it’s actually indirect dependencies that are the greatest potential source of vulnerabilities within applications. Research reported in the Datadog State of DevSecOps 2024 revealed that in Java, 63 percent of high and critical vulnerabilities derive from indirect dependencies. These vulnerabilities tend to be more challenging to identify because the additional libraries in which they appear are often introduced into an application secretly, without the development or security team’s knowledge.
Given these facts, it follows that a better approach to vulnerability remediation would take stock of an application’s complete set of direct and indirect dependencies.
OSV Guided Remediation: a step in the right direction
OSV is a Google-supported open source project that offers a vulnerability database and triage service. In April 2024, OSV launched the highly publicized Guided Remediation feature for the OSV-Scanner tool. OSV-Scanner offers the Guided Remediation feature through a simple command (osv-scanner fix) that automatically updates project dependencies.
Guided Remediation addresses some of the biggest issues inherent in traditional approaches to vulnerability remediation. OSV Guided Remediation uses source code to resolve the dependency tree and identify remediation paths. Using OSV’s own datasets to detect vulnerabilities in both direct and indirect dependencies, it recommends a remediation plan that reduces the likelihood of introducing new breaking changes. This plan groups vulnerabilities into actionable issues, which reduces the overall number of remediation steps by up to 70 percent.
While these advantages are significant, the Guided Remediation feature for OSV-Scanner also has important drawbacks. One limitation of the feature is that it currently works only with Node.js. In addition, since Guided Remediation operates only at the source code level, it cannot discern when a library might be hidden, or when it might be included in the codebase but never actually used in runtime. This hampers the tool’s ability to accurately determine which updates are optimal in your particular environment. Finally, Guided Remediation draws only upon OSV data to detect vulnerabilities. This potentially limits its ability to see known vulnerabilities that are published elsewhere.
Introducing Datadog’s smart vulnerability remediation
Datadog’s vulnerability remediation, which is found in our SCA and CSM products, optimizes the actions users should take to address vulnerabilities. As with OSV Guided Remediation, our approach aims to mitigate risks by recommending an update strategy that accounts for an application’s entire dependency tree. However, Datadog’s vulnerability remediation significantly improves upon the OSV tool’s capabilities in important areas such as runtime detection and visibility, language support, and data sources.
Runtime detection and visibility
Vulnerability scanners usually look for dependencies in package definition files, such as package-lock.json used by OSV, that are found in source code. Although looking directly at source code helps you construct an application’s dependency tree in a straightforward way, this method does not provide any visibility into the dependencies actually used during runtime.
Datadog, in contrast, detects vulnerabilities both in source code and—uniquely—in the runtime environment, whether they are introduced during execution, packaging, or deployment. Having visibility into the real components present at runtime gives a more accurate picture about which third-party dependencies are truly present and which might be hidden if you look only at source code. It also allows Datadog to prioritize vulnerabilities based on actual exploitability rather than theoretical risk. For example, our independent research reveals that only about 15 percent of the vulnerable components within an application’s dependency tree are actively used, which suggests that many vulnerabilities flagged in traditional scans might not really pose a threat.
Language support
Whereas (at the time of this writing) OSV Guided Remediation works only with Node.js, Datadog’s vulnerability remediation already supports seven major programming languages: Java, Node.js, .NET, Python, Go, Ruby, and PHP.
Additional data feeds
Unlike OSV Remediation, which relies solely on OSV Advisory definitions, Datadog uses vulnerability feeds from several different sources and creates its remediation plans based on all of them. These sources include ones that are publicly available—such as OSV, National Vulnerability Database (NVD), and GitHub advisories—as well as internal resources such as GuardDog (Datadog’s open source malware detection tool) and feeds from the Datadog Security Research Team. This wider range of source material allows Datadog’s vulnerability remediation to detect more flaws and malware in third-party code.
Coming soon: infrastructure scanning
At Datadog, we are actively developing solutions for vulnerability scanning and remediation guidance for infrastructure components, including containers and base images. This will make life even easier for security teams by broadening the range of assets that can be protected by Datadog’s vulnerability remediation capabilities.
Remediate vulnerabilities more effectively with Datadog
Remediating vulnerabilities on a case-by-case basis has proven to be inefficient, ineffective, and even counterproductive. While OSV Guided Remediation offers a better approach by taking into account vulnerabilities within an application’s entire dependency tree, it too has some significant limitations.
Datadog’s smarter vulnerability remediation offers significant improvements over these approaches through its ability to detect runtime usage of libraries, its broader support for different languages, and its more expansive base of source material about vulnerabilities. Together, these features enable a smarter, more efficient approach to vulnerability remediation that significantly benefits organizations aiming to better protect their applications and improve their security posture.
For more information about vulnerability detection and remediation in Datadog, see our documentation. And if you’re not yet a Datadog customer, you can sign up for a 14-day free trial to get started.
