Almost all tasks within a Linux system, whether it’s an application, system daemon, or certain types of user activity, are executed by one or more processes. This means that monitoring processes is key to detecting potentially malicious activity in your systems, such as the creation of unexpected web shells or other utilities. In this post, we’ll look at:
- how understanding the Linux process tree can help you identify security threats
- what process information can help you determine the scope of a breach
We’ll also look at how Datadog Cloud Workload Security can help you monitor processes across your entire environment to surface security threats.
A primer on the process tree
In Linux, each process is generated by a preceding parent process and can generate one or more child processes. Following this parent/child structure, active processes form a process tree that starts with the systemd process that runs when Linux first boots and ends with the most recently generated processes. This parent/child structure is particularly useful for revealing security threats because, unlike simple indicators of compromise (IOCs) like adversarial IP addresses or file hashes, it’s difficult to fake or change. For instance, while attackers can change an IP address, it’s much harder to hide that an application has spawned a new suspicious child process.
Identify suspicious processes
As you monitor your applications, it’s important to look out for any launched shells or utilities (e.g., bash or curl) that are children of key processes in your environment. For example, if a Java application process generates a shell you aren’t anticipating, it could indicate that a malicious actor has infiltrated your app and launched a web shell attack. When these attacks are successful, they can create backdoors to your infrastructure that allow attackers to access sensitive data and execute commands without authorization.
Likewise, you should check if a process spawned utilities like nmap, which an attacker can use to survey your network for further vulnerabilities to exploit, or passwd, which can be used to change user passwords and grant attackers higher privileges.
Once you’ve identified a malicious process, it’s important to investigate the scope of the attack, including what information the attacker has potentially gained access to.
Use process data to determine the scope of an attack
Linux processes include metadata that can help you determine the scope of an attack. The key types of information to look at are:
- environment variables
- command-line arguments
Environment variables
Due to their relationship, child processes inherit access to any environment variables available to the parent process. Though it’s not considered best practice, environment variables are often used to store sensitive data like API and GitHub keys, or even database credentials. Using our example of an attacker successfully creating a web shell from a Java application process, the shell would be able to see any environment variables associated with your application process, such as database credentials for a SQL server containing sensitive customer data.
Examining what environment variables a process includes can help you determine the full scope of a threat. You can quickly view a list of a process’s environment variables by using the Linux command ps faux to get its PID and then running the following:
cat /proc/<PROCESS_PID>/environ Please note, however, that this detection method only works while the process is running. This makes it challenging to view potential attacks without a continuous monitoring tool.
Command-line arguments
In addition to environment variables, it’s important to know what command-line arguments an attacker used when starting a malicious process. Command-line arguments hold information that’s critical to determining the nature of a security threat. Arguments may include identifying data like the IP address an attacker used when they issued a curl command to download a malicious payload, as well as activity data like encoded Python scripts that were run directly in the command line. Insight into a process’s command-line arguments can help you view what occurred during an attack so you can plan how to respond.
For a quick look at a process’s command line arguments, get its PID and run a command like the following:
ps -p <PROCESS_PID> -o argsThis approach, however, is limited because you need to be able to catch the process before it terminates. Next, we’ll look at how Datadog’s Cloud Workload Security helps you detect attacks and view relevant metadata.
Detect threats in your Linux processes with Datadog Cloud Workload Security
Datadog Cloud Workload Security (CWS) analyzes the full process tree across all your Linux hosts and containers in real time to automatically detect the kind of threats we’ve looked at. Datadog includes out-of-the-box workload threat detection rules that help you immediately respond to potential security threats by flagging suspicious behavior like the execution of passwd and nmap utilities.
In addition to out-of-the-box workload security rules, Datadog enables you to write your own custom rules. Security rules are constructed with Agent Expressions that enable you to define what process activity to look for with as much specificity as you want. For example, you can instruct Datadog to detect if a Java process generated a bash shell, or even watch for processes run with specific command-line arguments. To learn more about how to construct your own custom workload security rules, check out our documentation.
If Datadog detects any processes that match a rule, it will generate a Security Signal. Security Signals include full context around the suspicious process, including environment variable keys (without collecting the associated values), command-line arguments, and other metadata. You can use this information to quickly determine the scope of an attack before planning your response.
Start today
No matter which distribution you work with, processes are at the heart of any Linux system. In this post, we looked at how understanding processes and their relationships can help you spot suspicious behavior and determine the severity of a security breach. Datadog Cloud Workload Security monitors process activity throughout your infrastructure at the kernel level in real time to reveal any suspicious or malicious behavior. Read our documentation to learn more or sign up today for a 14-day free trial.
