Many organizations rely on distributed tracing in Datadog APM to gain end-to-end visibility into the performance of their Kubernetes applications. But as teams grow, it can become impractical for them to manually configure each new application with the libraries and environment variables needed for tracing. Traditionally, this manual configuration has required developers to build a new application image with the appropriate tracing library installed—a process that activates a CI/CD pipeline and brings development to a temporary halt.
To solve this problem, we’ve developed a workflow that automates instrumentation, enabling you to configure tracing for your Kubernetes application without changing or rebuilding your application image. You can now inject Java, Node.js, and Python tracing libraries directly into your pod at runtime via the Datadog Admission Controller by writing a single annotation into your application’s YAML file.
In this post, we’ll cover the following topics to help you quickly set up Datadog APM for your Kubernetes environment without needing to rebuild or redeploy your application image:
- Faster time-to-value for APM with library injection
- How to instrument your application with library injection
Faster time-to-value for APM with library injection
Previously, configuring your applications to send traces required building a new application image that included the library and updated configuration files. This manual method of instrumentation is lightweight when dealing with a small number of applications. However, it can amount to greater operational overhead as platforms continue to grow with new applications and features that require tracing.
Library injection via the Datadog Admission Controller enables you to configure tracing for your Kubernetes application without building a new application image. You can enable library injection by adding a single annotation in a pod’s configuration template when you configure the Datadog Agent for your Kubernetes cluster. Once you apply this new configuration, Datadog will automatically detect the annotation and inject the appropriate tracing library into your pod at runtime as a separate init container. After it’s deployed, the container will copy relevant library files from the file system to a shared path accessible by all your application containers. Because the tracing library is orchestrated as a separate container, this method does not require you to change your application code or push a new container image. This avoids the CI/CD pipeline that would launch and make you wait for your new application image to build, test, and then redeploy. As a result, you can begin collecting traces from your Kubernetes applications in just minutes.
The Admission Controller can also be used to inject environment variables and standard tags such as env, service, and version from application labels into your container tags. Because library injection relies on pod annotations, you can choose to adopt it into your instrumentation workflow or continue to send traces from your application to the Agent by using UDS or TCP ports.
How to instrument your application with library injection
To begin collecting traces with Datadog, first make sure that you’re running the Datadog Agent V7.40+ and Datadog Cluster Agent in your Kubernetes environment. Instrumenting your application through library injection then involves enabling the Admission Controller, configuring the necessary labels and annotations, and enabling unified service tags for the associated pod.
Enable the Admission Controller
If you’re managing Kubernetes with Datadog’s Helm chart (v2.35.0+), the Admission Controller is enabled by default, and you can proceed to the next step. If you’re using the Datadog Operator instead, you can follow these instructions to enable the Admission Controller for the Datadog Agent.
Annotating your pod with the correct tracing library
Once the Admission Controller is enabled, Datadog will apply changes to your pod at runtime and inject the appropriate library into a separate container. You can perform this step and annotate your pods for library injection by using the labels and annotations shown below:
...
template:
metadata:
labels:
admission.datadoghq.com/enabled: "true"
#Enable Admission Controller to mutate new pods in this deployment
admission.datadoghq.com/config.mode: "socket"
annotations:
admission.datadoghq.com/<LANGUAGE>-lib.version: "<VERSION>"
#Enable library injection for a specified tracing library.
...Configuring unified service tagging
With the Admission Controller, you can also quickly configure your application for unified service tagging. Unified service tags enable you to identify meaningful trends by automatically cross-referencing your traces, logs, and metrics through the common reserved tags env, service, and version. Using these tags, you can quickly filter your traces to troubleshoot a new version of your application, or aggregate your services to identify those with the largest customer throughput. You can easily set unified service tags for your application by including the following labels in both your deployment metadata and pod template.
...
metadata:
labels:
tags.datadoghq.com/env: "<ENV>"
tags.datadoghq.com/service: "<SERVICE>"
tags.datadoghq.com/version: "<VERSION>"
...Once you’ve finished configuring your deployment object, your pods will be automatically instrumented. Because Datadog uses a dedicated container for the tracing library injected named datadog-lib-init, you can check to see if your injection was successful by verifying that the pod includes that container, as shown below:
Get started with Datadog today
Library injection via the Admission Controller simplifies the instrumentation process, enabling you to view APM traces in just minutes without changing or rebuilding your application. More information about library injection can be found in our documentation. You can learn more about monitoring your Kubernetes application with Datadog in this blog post.
If you don’t already have a Datadog account, you can sign up for a free 14-day trial today.
