Engineering the Datadog Agent for FedRAMP High® Certification

Software that runs inside customer-managed infrastructure creates a particular challenge at the FedRAMP High baseline. It still has to meet the applicable security and compliance requirements, even though the vendor does not control the surrounding operating system, libraries, network configuration, or maintenance practices. For Datadog, that challenge centers on the Datadog Agent, which runs directly on customer-managed hosts to collect logs, metrics, traces, and security signals.

The scale of those requirements is substantial. The current Rev. 5 FedRAMP High baseline defines more than 400 controls that have to be implemented, documented, and supported with evidence over time. Meeting those requirements involved significant engineering and operational work, along with ongoing responsibility for maintaining compliance. That commitment also helps explain why organizations operating at this level often have fewer options for critical capabilities such as observability and security monitoring.

Meeting that bar required more than extending the architecture we had developed for FedRAMP Moderate. The Agent needed to use the required cryptography across its Go and Python runtimes, remain within clearly defined cryptographic boundaries, and fail predictably if those requirements could not be met. Those guarantees also needed to be maintained as customer environments and the Agent itself changed over time.

We already had a practical way to protect outbound Agent traffic in FedRAMP Moderate environments. But as we evaluated that design against the High baseline, it became clear that an external proxy could not provide the necessary coverage on its own. The resulting work changed where cryptographic enforcement lived in the Agent and how we built, tested, and maintained the software around it.

Now that Datadog for Government has achieved FedRAMP High certification, this post explains how we approached these engineering challenges. The resulting Agent architecture helps customers integrate the Datadog Observability Platform with minimal friction while meeting stringent security, operational, and compliance requirements.

Why FedRAMP High required a different approach

The differences between the Moderate and High baselines extend beyond the number of controls. High does not allow the same exceptions as Moderate and places stricter expectations on how controls are enforced and how organizations demonstrate that they remain effective. It also requires the system’s cryptographic modules to satisfy SC-13 Cryptographic Protection requirements at all times. For example, cryptographic behavior needs to be explicitly defined and testable. Supporting an approved cryptographic configuration is not enough if software can start or continue operating outside it.

Relevant boundaries also need to be clear. The FedRAMP boundary identifies the system covered by the certification. The cryptographic boundary is defined by the security policy for the validated FIPS module, not by a general rule about whether an operation occurs on a host or within a particular process.

Failure modes face similar scrutiny. Both FedRAMP Moderate and High require organizations to document and understand them, but High introduces additional requirements for validation and adherence to cryptographic boundaries. For customer-installed software, that makes deterministic configuration, fail-closed startup checks, and auditable runtime behavior particularly important.

Those expectations continue after certification. High raises the operational bar through stronger evidence expectations, continuous monitoring, and tighter change management. Controls must remain effective through security patches, dependency updates, platform changes, and new product releases. Organizations must maintain supporting evidence as systems evolve.

Meeting these requirements is difficult enough within the infrastructure a vendor controls. The Datadog Agent has to continue meeting them across customer environments that may use different operating systems, CPU architectures, library versions, and network configurations. Those environments can also drift as customers install updates or change how their systems are managed.

The FedRAMP Moderate architecture

In January 2022, we released FIPS Proxy as part of our FedRAMP Moderate solution. The design provided secure transport and hardened defaults without requiring major changes to the Datadog Agent itself. In that architecture, the Agent sent outbound traffic through a local re-encrypting proxy. The proxy applied FIPS-validated cryptography before connections left the host.

The original implementation used a validated OpenSSL 1.0-era module derived from the OpenSSL FIPS Object Module 2.0 family. We later upgraded the proxy to use a validated OpenSSL 3 provider. This gave the Agent a clear control point for outbound TLS connections. However, it did not make the Agent itself a validated cryptographic module. Cryptographic operations within the Agent remained outside the proxy’s scope.

Although we could potentially have worked around some of the available tooling limitations, their combined risks led us to use the proxy-based architecture. Customers could continue using the standard Agent, while Datadog isolated the FIPS-specific changes in a separate component. That made the solution available sooner and avoided rebuilding the Agent around cryptographic tooling that did not yet support our full platform footprint.

Why FIPS Proxy fell short

The proxy remained a useful architecture for FedRAMP Moderate, but extending it to meet the High baseline would have created several coverage and maintenance problems. The proxy needed to know which outbound endpoints to intercept when the Agent added a new destination.

That created a risk that a new connection could be added to the Agent but omitted from the proxy. The security guarantee depended on two configurations remaining synchronized. A High-assurance design needed to protect new connections by default rather than relying on a separate configuration step.

The Agent also includes an embedded Python runtime for Datadog integrations and customer-written checks. Those integrations can bring their own dependencies and perform cryptographic operations inside the Python process. 

The proxy could protect traffic after it left that process, but it could not enforce which cryptographic provider the Python code and its dependencies used internally. Accounting for every customer integration and dependency through proxy configuration was not practical.

FIPS Proxy focused on outbound traffic. Applying the same model to incoming Agent connections would have required each entry point to be enumerated in HAProxy and maintained as the Agent evolved. That would have introduced another configuration that could drift out of sync with the product.

Separately, the OpenSSL-based cryptographic module behind the original proxy was nearing the end of its validation lifecycle. At the time we began designing the High solution, the newer OpenSSL 3 FIPS module was not yet both validated and compatible with the existing HAProxy architecture.

We later upgraded FIPS Proxy to use the validated OpenSSL 3.0.8 provider. That kept the Moderate solution current, but it did not address the broader coverage limitations of the proxy model. The most significant issue was scope. FedRAMP High required us to account for more than proxy-wrapped TLS connections.

Cryptographic operations inside the Agent, interactions between its runtimes, ingress traffic, and failure behavior all needed more explicit and auditable treatment. An external proxy could not enforce those requirements within the processes performing the operations.footprint.

How the cryptographic ecosystem changed

When we first designed this system, Go did not provide a natural way to plug in a FIPS-capable cryptographic module. That differed from ecosystems such as Python, Java, and C, which had more established mechanisms for binding applications to validated providers. The available approaches for Go each introduced tradeoffs, including limited operating-system or CPU architecture coverage and uncertainty about how the underlying FIPS component would be maintained and updated over time.

By the time we revisited the architecture for FedRAMP High, several underlying technologies had matured. The OpenSSL 3.0.0 FIPS provider had been validated and was better suited to direct application integration. The OpenSSL 3.0.8 provider was later validated with fewer technical restrictions.

Go’s BoringCrypto implementation moved into the main Go repository and added ARM64 build support. Microsoft also developed a Go distribution that matured into a documented path for using operating-system cryptographic libraries, including OpenSSL on Linux, Cryptography API: Next Generation on Windows (CNG), and later CommonCrypto and CryptoKit on macOS.

Python could now be built against OpenSSL 3 as well. This made it possible for the Agent’s Go and Python runtimes to use the same installed OpenSSL library and FIPS provider on Linux. Together, these changes made an in-process architecture more viable than it had been when Datadog built FIPS Proxy.

Choosing between BoringCrypto and Microsoft Go

After evaluating the available options, we narrowed the options for the Agent’s main Go runtime to BoringCrypto and Microsoft’s Go distribution.

BoringCrypto offered the more established route. It worked with familiar Go build patterns and would have provided strong coverage for the AMD64 Linux systems common in cloud environments. Its limitations became more important when we evaluated it against the Agent’s complete platform footprint. Going this direction would have given us less direct control over patching the underlying cryptographic library. The available implementation also lacked some required capabilities, including TLS 1.3 support at the time of the evaluation. Windows and other supported architectures would have presented additional challenges.

BoringCrypto would have made the initial implementation easier, but the resulting FIPS Agent would have supported a narrower range of customer environments. Microsoft’s Go distribution provided broader platform support and allowed the Agent to use OpenSSL on Linux and CNG on Windows. It also gave us more control over the cryptographic libraries and their patching lifecycle. On Linux, it allowed the Agent’s Go runtime to share the same OpenSSL installation as the Agent’s embedded Python runtime. Instead of maintaining separate cryptographic implementations for Go and Python, both could use the same validated provider.

Microsoft’s implementation was newer and less widely adopted than BoringCrypto, which created more uncertainty around long-term support. Choosing it also meant that we might need to maintain parts of the integration directly. That maintenance responsibility mattered because the cryptographic component needed a reliable path for security updates and continued validation support.  We accepted the additional engineering risk rather than limit the FIPS Agent to a smaller set of environments. 

Building the FIPS Agent

Selecting the Go implementation addressed only one part of the problem. We still needed to change the Agent’s builds, Python runtime, dependencies, and testing infrastructure. One early question was whether every Agent should use the FIPS-capable Go runtime. Performance testing showed approximately a 2% degradation when running the Microsoft Go build. The difference was small, but applying it universally would have introduced a regression for customers who didn’t need FIPS support.

We decided to maintain separate FIPS and non-FIPS builds. That let customers who needed FIPS support use the FIPS build without changing the performance characteristics of the standard Agent.

The Python changes required substantially more work. We rebuilt the embedded Python runtime and several of its main dependencies, including its cryptography libraries, to use the installation-level OpenSSL library and validated provider.

The work spanned multiple teams and several interdependent build components. Python and its dependencies needed to use the shared provider consistently after the Agent was installed, rather than bringing separate OpenSSL implementations into the package.

Other Agent components also used cryptographic operations that were not permitted in the target configuration. Teams needed to identify and replace uses of weak hashing algorithms, nonstandard key-generation methods, incompatible entropy sources, and other disallowed primitives. These changes extended beyond the central Agent codebase. Each contributing team needed to update its component before we could assemble and validate the final artifacts.

A test harness for the installed Agent

We also built a repeatable test harness to verify the completed artifacts. Compiling the Agent against the intended libraries didn’t prove that it would use the correct cryptographic backend after installation. We needed additional testing to confirm that Go operations were forwarded to OpenSSL or CNG as expected and that the Agent responded correctly when the required provider was unavailable. That let us verify the Agent’s runtime behavior instead of relying on build configuration alone.

The final FIPS Agent brought the Go and Python runtime changes together with updates across individual components. The complete package then had to pass runtime validation after installation.

The FIPS Agent architecture

The new architecture still relies on established platform libraries for cryptographic operations. The main change is that those libraries are now integrated into the relevant Agent runtimes rather than operating behind an external proxy.

On Linux, both Go and Python use the installed OpenSSL library and its validated FIPS provider. On Windows, the Go runtime uses CNG.

This brings the validated cryptographic capability closer to the code performing the operations and removes an additional network hop within the host. New endpoints no longer need a matching proxy rule before receiving the expected protection. Cryptographic enforcement now occurs within the runtime rather than through a separately maintained list of connections.

Sharing the OpenSSL installation between Go and Python on Linux also simplifies updates. We can patch or upgrade the common provider without reconciling multiple separately packaged copies of the library.

Startup checks and failure behavior are easier to define as well. If the Agent cannot establish the required cryptographic configuration, it can fail before beginning normal operation instead of continuing in an unexpected mode.

What the redesign changed for customers

The work behind the FIPS Agent is complex, but customers interact with a simpler installation. FIPS Proxy added another component that needed to be deployed, configured, monitored, and updated. It also introduced a separate set of endpoint rules that had to remain aligned with the Agent.

The FIPS Agent brings that enforcement into the product itself. Its installation and day-to-day operation more closely resemble those of the standard Agent, while the FIPS-specific behavior remains part of the build. That makes common security review questions easier to answer. Teams can identify which cryptographic providers the Agent uses, where they apply, and whether the required cryptography is enforced and how the Agent prevents operation outside the expected mode. They can also review how we test updates and manage changes to the underlying libraries. The architecture also makes it easier to determine what data leaves the environment, how that data is protected in transit, and what evidence supports ongoing security operations.

The architecture does not remove the work required for FedRAMP High. It reduces the amount of environment-specific configuration customers need to maintain and makes the resulting behavior easier to demonstrate during an audit.

The Agent was only one part of the work

The FIPS Agent represents one part of the broader effort required for Datadog for Government to achieve FedRAMP High certification. Other work included artifact signing, artifact validation, provenance, reporting, and the operational controls required to maintain evidence over time. Those areas brought their own design and implementation challenges.

The Agent redesign shows how deeply the High baseline can affect a product that runs inside customer infrastructure. The final architecture was not the result of replacing one dependency or enabling a new configuration. It required coordinated changes across runtimes, build systems, product components, and release processes

Maintaining the architecture after certification

FedRAMP High certification does not end our work. The Datadog Agent needs to continue meeting FedRAMP High requirements as operating systems, dependencies, cryptographic standards, and customer environments change. That makes patching and future platform support part of the compliance design. The cryptographic implementation needs a viable update path, and changes to the Agent must preserve the runtime guarantees established by the FIPS build.

We are also evaluating newer upstream Go support for FIPS 140-3-validated cryptography. That work may offer future performance or deployment benefits, but it does not provide an immediate replacement for the current architecture. The embedded Python runtime would still need to use OpenSSL, so any change would need to account for both runtimes.

For installation and configuration guidance, see the FIPS Agent documentation. Teams can also request a demo or start a free trial for Datadog for Government at the FedRAMP High baseline.