Ingest OpenTelemetry logs with the Datadog Agent

OpenTelemetry (OTel) is an open-source, vendor-neutral observability solution that provides a suite of components—including APIs, SDKs, and a data collector—that enable teams to collect and communicate telemetry data from cloud-native applications and services. OTel also defines the OpenTelemetry Protocol (OTLP), a standard for the encoding and transfer of telemetry data. Organizations can use OTel to collect and export observability data from their environment to their preferred backend, such as Datadog.

Our Datadog Agent already supports the ingestion of OTLP traces and metrics from all OTel-instrumented applications. Now, we are proud to announce support for the ingestion of OTLP logs as well. Teams that deploy the Agent to observe their OTel applications can now enjoy out-of-the-box support for all the features of our log processing pipelines—including automatic log parsing, enrichment, and trace-log correlation—all without needing to further update code.

Previously, clients using Datadog to ingest OTLP logs had to follow some workarounds. One option was to write raw OTLP logs to a file on disk, and then have the Agent track those files and read them in. Another option was to run, separately from the Agent, an OTel Collector instance configured with the Datadog Exporter. In this latter setup, the Datadog Exporter would be used to ingest only OTLP logs, while metrics, traces, and non-OTLP logs went through the Agent. However, these options were more complex because they did not allow all telemetry signals to be ingested by the same mechanism—resulting in a dual pipeline that increased the operational overhead for teams. In addition, neither workaround offered the advanced features for log collection that come with the Datadog Agent, such as log filtering, data scrubbing, and multi-line aggregation.

In this blog post, we’ll go over how you can configure your Datadog Agent to collect OTLP logs from applications that have been instrumented using the OTel SDKs.

Sending OTel logs to the Datadog Agent

To collect and transmit OTel logs, you’ll first need to enable the SDK for your application’s language by using either OTel’s automatic instrumentation or manual instrumentation. The OTel Log SDK captures logs sent out by your logging framework and then generates new logs in OTLP format. It can be configured to transmit these OTLP logs via HTTP or gRPC to two endpoints. (By default, these endpoints are two ports on the local machine.)

The next step is to configure the Datadog Agent to listen for OTLP logs sent to these endpoints, as described in our documentation. This configuration allows a running Agent instance to receive the logs in OTel format, enrich them with Datadog-specific fields, and forward them to the Datadog backend. From there, they can be viewed and analyzed in Datadog together with the rest of your telemetry.

The code snippet below is an example of an otlp_config section that can be added to any Datadog Agent configuration file to enable OTel log ingestion. Note that, to protect you from unexpected log usage that may impact billing, all log collections are disabled by default. Therefore, to enable OTLP log ingestion, you first need to explicitly enable log collection as a whole through the top-level setting logs_enabled:true. Then you can enable OTLP log ingestion by setting otlp_config.logs.enabled to true, as follows:

logs_enabled: true
otlp_config:
  receiver:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318
  logs:
    enabled: true

Alternatively, you can configure OTLP log ingestion by setting the environment variable DD_OTLP_CONFIG_LOGS_ENABLED to true. Note that with this method, it is still required to enable log collection in the Agent by setting DD_LOGS_ENABLED to true.

Correlating traces and logs

Once your Agent has collected logs, you can view them in the Log Explorer. For each log, the Agent automatically injects any associated trace_id values that are present in the generated OTLP logs. The platform then reads the trace_id in each log and (if the trace is also sent to Datadog) automatically links to the associated trace, allowing you to quickly get context for your logs and troubleshoot more effectively.

Similarly, on the APM trace view page, you can find the log that is correlated to this trace. This cross-correlation allows you to quickly locate important debugging information from logs when you see a trace revealing high latency, errors, or other unexpected behaviors.

Benefits of Agent OTel log ingestion

For teams that already use the Agent as part of the workflow to ingest OTel data, configuring the Agent to ingest OTLP logs offers a number of benefits:

• Unified telemetry ingestion: By enabling the Agent to ingest OTel log data along with OTel metrics and traces, teams can now consolidate the collection of all OTel data through a single mechanism. This unified approach simplifies the telemetry data collection process and eliminates the need for workarounds, reducing administrative overhead and streamlining the onboarding process for engineering teams.
• Enhanced observability: With out-of-the-box trace-log correlation, users can seamlessly link OTel logs with other OTel data, including traces and metrics, without any extra code changes. This helps provide a more complete view of an application’s performance, making it easier to detect, diagnose, and resolve issues.
• Robust re-trying logic: The Datadog Agent is well-equipped to handle network interruptions. If communication to the Datadog backend is interrupted, the Agent employs a robust retry mechanism, which reduces the risk of log data loss and improves the resiliency of log data collection.
• Advanced log processing: The Datadog Agent offers sophisticated log processing capabilities through features such as the ability to filter or exclude specific logs, scrub sensitive data, and automatically aggregate multi-line logs. These advanced capabilities help ensure that the log data ingested is both highly secure and consumable.
• Optimized network costs: The Agent employs efficient batching and compression techniques, ensuring that the telemetry data is transmitted in a compact form. This reduces the amount of data sent over the network and helps minimize associated network costs.
• Ongoing support and performance improvements: By connecting OTLP log ingestion to the Datadog Agent, organizations can take advantage of Datadog’s robust enterprise support for the Agent. And as the Agent is continually upgraded for performance improvements, they can also realize those performance benefits for log ingestion in an ongoing way.

Taken together, these benefits enable teams to implement OTel in a simple way that fits seamlessly within their existing Datadog workflows—all while optimizing costs and functionality. By drawing upon the Agent’s native data ingestion for OTel log data, teams can tap more fully into Datadog’s powerful capabilities that, when combined with metrics and traces, provide visibility into application performance that is both deep and comprehensive.

Enhance your observability today

Check out our documentation to learn more about collecting logs from OTel-instrumented apps in Datadog. If you’re not already a Datadog customer, sign up for a 14-day free trial today.