Monitoring the health of your infrastructure and services requires you to understand the performance of fundamental system processes. But particularly in large environments, the sheer volume of processes can make their performance and resource usage difficult to track, let alone troubleshoot. To further complicate matters, it can be challenging to gather the necessary context on your processes to effectively categorize and monitor them—historically, it’s only been possible to identify processes using tags that come from the host they’re running on, either directly or through elements of the associated cloud infrastructure, such as containers, pods, and other resources. An individual process’s command line contains information on its associated groups, services, and configuration, but this metadata is not always parsed as facets or metrics in your observability platform.
Datadog Live Processes now includes Process Tag Rules, which enable you to enrich your system’s end-to-end visibility by deriving tags from a process’s command line, and to create performance metrics and alerts specific to your processes’ tags.
In this blog, we’ll explore how Processes Tag Rules help enhance troubleshooting and monitoring your host processes by enabling you to:
- Quickly find common issues with groups of processes
- Track system health by creating metrics and monitors on your custom process groups
Quickly find common issues with groups of processes
A process or a group of processes can represent vital application- and system-level operations. However, it can be difficult to obtain visibility into the data that will tell you how to group your processes for more effective monitoring.
Process Tag Rules enable you to discover what processes are related to each other using information embedded in the command line configuration of your processes. By creating these tags, you can monitor performance and resource utilization for groups of related processes, giving you visibility into system-wide performance to help you troubleshoot issues.
You can create a new Process Tag Rule from the Manage Process Tags tab in Datadog Live Processes.
After you click on New Process Tag Rule, you can configure your rule, starting with name, scope, and expression. Rule expressions are where you define your tags based on the command line configuration of your processes, using the standard Grok language syntax.
For example, you can configure a Process Tag Rule that matches the command line for tini processes in a particular environment and extracts the command line flags as tags. You can filter your processes down to command:tini and a specific cluster_name to see how many processes your rule will apply to. The next step is to create an expression for the rule—in the screenshot below, this is defined as tini -- %{notSpace:tini_start_process_name} %{notSpace:tini_sub_process_name}. This rule will capture the two tokens following tini -- as new tags that define the facets tini_start_process_name and tini_sub_process_name.
You can test your new rule by providing a sample of a command line, such as tini -- bash startstuff.sh. If the rule is validated, you’ll see example tags from your command sample, so you can make sure these new tags meet your expectations.
After you click Create Rule to save the new Process Tag Rule, it will be applied to all matching processes, and your new tags will be available to search from the Processes view. To view process groups with these newly generated tags, you can group your processes by the tag group you just created—in our case, tini_start_process_name.
You can also use Process Tag Rules to group processes by service in certain scenarios. For example, with the shell svchost Windows system process, the -s command line option will load the specific service that the process is running on. With a Process Tag Rule, you can define this value to be extracted as a tag.
Track system health by creating metrics and monitors on your custom process groups
Creating new tags also enables you to create metrics based on the process groups you define. You can create a new metric right from the Manage Metrics tab by clicking New Metric.
From there, you can provide the metric dimensions you want to collect for the process groups you’re interested in.
For more information on configuring Process Metrics, check out our documentation.
You can also create Live Process Monitors scoped to your rule-based tags—simply choose the New Monitor option from the New Metric dropdown in your Processes view.
From here, you can create a Live Process Monitor that will show you whether processes tagged based on your custom rules are running. For example, using the svchost_service tagging rule we created early, you can create a monitor on the svchost_service:WinRM process group that ensures that Windows Remote Management is running across your fleet of Windows hosts.
You can also set metric monitors based on the metrics you create for your custom process groups. For example, you may want to be alerted when certain groups of processes reach specific thresholds for CPU and RSS memory usage, as high values for these metrics might signal that adjustments are required for process replicas running for a particular service.
Enhance your process-level observability
Datadog’s Process Tag Rules expand your organization’s ability to troubleshoot and monitor your process-level data. By proactively monitoring and addressing these issues early, you can safeguard mission-critical services, optimize resource allocation, and build resilience into your systems.
Process Tag Rules are available if you are a Datadog customer. If you’re not already using Datadog, you can start today with a 14-day free trial.
