Health and performance issues are easier to understand—and to troubleshoot—when you can use tags to aggregate your data across many overlapping scopes. But while some scopes come directly from your infrastructure, others are constantly evolving to reflect the needs of your product or organization. You can only track your data effectively if you can define—and redefine—your scopes on the fly.
The solution is to combine your existing tags into complex expressions using Boolean logic. With this technique, a datapoint belongs to a scope if—and only if—it satisfies a set of expressions about other scopes: in EU regions but not US ones; in your development and staging environments but not production; and so on. You can use Boolean operators in Datadog metric queries, giving you a more concise, meaningful way to define the scope of your metrics, enabling you to quickly get the visibility you need into complex environments.
How Boolean-filtered metric queries work
You can define queries to scope Datadog metrics within dashboards and notebooks. Now, you can build even more powerful metric queries by using the following Boolean operators to establish relationships between tags:
ANDORNOT(or the alias!)
You can also group Boolean operations using parentheses, and use the IN operator to scope your query to any tag key within a list of key:value tags. For example, if you wanted to view the average number of running Kubernetes pods managed by three teams within your data processing org, you could write the following in the metric query editor:
avg:kubernetes.pods.running {team IN (ingestion, storage, query)} by {team} Capture complex scopes
The requirements of any software project are constantly evolving, which means that you often need to define new categories to get the visibility you need into your infrastructure. For example, you might be investigating unexpectedly low message throughput in RabbitMQ queues (rabbitmq.queue.messages.rate) related to your shopping cart analytics pipeline. You could focus your investigation by using as specific a filter as possible, e.g., service or env, and group the metric by rabbitmq_queue to examine each queue individually—which runs the risk of displaying queues that are not relevant to your search. Or you could define a separate query for each value of the rabbitmq_queue tag that you want to investigate.
You can now write your query more easily—and more elegantly—by using an expression similar to the following to view only the queues you need to investigate:
avg:rabbitmq.queue.messages.rate {rabbitmq_queue IN (carts_raw, carts_items, carts_initialized)} by {rabbitmq_queue}More informed leadership
Engineering managers need the most revealing team-by-team data to understand how to set priorities and lead projects. As microservices-based architectures become more popular, organizations have given small teams ownership over individual services (e.g., AWS’s “two-pizza teams”). In this context, addressing complex incidents can require teams to collaborate in an ad hoc fashion.
Using Boolean-filtered metric queries, managers can quickly and easily get on-the-fly answers to any question about teams within their organization. For example, a manager who wants to view metrics for three teams in a six-team department can use the OR operator to scope metrics to only those teams—without having to filter by the less precise department tag or set up separate metric queries for each team they want to track. Alternatively, if a manager knows that all teams in a department are focusing on a release except for one, they can use the team tag with the NOT operator to exclude the irrelevant team.
Monitoring IN (comprehensive, flexible, precise)
Boolean-filtered metric queries are now available in Datadog dashboard widgets and notebooks, meaning that you can quickly get the visibility you need into your infrastructure, no matter how complex the scope. Not yet a Datadog customer? Get started today with a free trial.
