Since StatsD was introduced by Etsy in 2011, it has become a mainstay of infrastructure monitoring. But as dev and ops teams rely more and more on containerized microservices, they have pushed the limits of StatsD’s design. One major shortcoming is that StatsD has no built-in support for tagging your metrics with key-value pairs. If you want to get context into different slices of your infrastructure but are using software that can only generate metrics in StatsD format, you’ll need some way to process your StatsD metrics into tags.
To make this task quicker and easier, Datadog has introduced DogStatsD Mapper, which automatically generates tags for StatsD metrics based on user-configured rules. By using DogStatsD Mapper, you can convert parts of your StatsD metric names into tags without having to re-instrument your code, and without having to run a separate server (like statsd exporter) to transform your metrics.
StatsD and tags
In a typical StatsD setup, application code is instrumented to send metrics as single-line datagrams to a StatsD server. StatsD then flushes these metrics periodically to a storage backend. The names of StatsD metrics are designed to work with Graphite’s hierarchical directory structure, and take the form of dot-delimited namespaces. Since there are no keys, only values, the only way to group or filter metrics by different combinations of metadata is to use a dedicated tag processing server such as Graphite’s Carbon.
Some tools generate metrics only as StatsD datagrams. Apache Airflow, for example, emits metrics using Python’s StatsD library. If parts of your infrastructure are instrumented to emit metrics in StatsD format, and you want to group and filter your metrics by tag, you’ll need a way to generate tags downstream from your StatsD client. The airflow-exporter plugin for Airflow attempts something like this, but exposes only two Airflow metrics.
How DogStatsD Mapper works
The Datadog Agent can ingest StatsD metrics as well as metrics in the DogStatsD format, which extends StatsD to include tags. DogStatsD Mapper runs as part of the Agent’s built-in DogStatsD server. When the Agent receives an untagged metric in StatsD format, it uses rules to map segments of the metric’s namespace to a list of tags. It then gives the metric a new name and processes it like any other tagged metric. (If you’re curious as to how this works behind the scenes, the Agent is open source.)
Tag any StatsD metric
To create a rule, specify the namespace of a StatsD metric, replacing all or part of each namespace segment with wildcards or regular expressions. Then indicate the name of the metric as it will appear within Datadog and where in the metric’s namespace DogStatsD Mapper will look for tags. If DogStatsD Mapper detects that the namespace of a StatsD metric matches a rule, it will extract tags and rename the metric.
For example, you can generate tags for your Airflow StatsD metrics by copying rules we provide in our documentation and pasting them into your Datadog Agent configuration file:
datadog.yaml
dogstatsd_mapper_profiles:
- name: airflow
prefix: "airflow."
mappings:
# [...]
- match: "airflow.dag.*.*.duration"
name: "airflow.dag.task.duration"
tags:
dag_id: "$1"
task_id: "$2"
# [...]Using this configuration, DogStatsD Mapper will transform any StatsD metric matching the pattern airflow.dag.*.*.duration into a metric called airflow.dag.task.duration with the dag_id tag set to the name of the Airflow DAG and the task_id tag set to the name of the Airflow task.
Start mapping your StatsD metrics
If you’re a Datadog customer, you can start defining mapping rules right away to extract tags from your StatsD metrics and get insight into different slices of your environment. With DogStatsD Mapper built into the Datadog Agent (version 7.17+), you can enjoy the full power of Datadog tagging with your own StatsD-instrumented legacy code as well as with infrastructure tools written with StatsD in mind (e.g., Airflow). If you’re new to Datadog, sign up for a 14-day free trial.
