For engineering teams who are investigating and resolving incidents together, it’s a common practice to use terminals alongside IDEs and web UIs such as Datadog and the AWS Console. But communicating progress that’s made in terminals and maintaining a clear record of past activities is a manual, cumbersome process. Therefore, incident responders struggle to stay on top of exchanges concerning root causes—such as which hypotheses have been explored already, and which of these have turned out to be true or false. The lack of access to a record of activity in terminals also makes it harder for teams to learn from the most experienced responders about how best to mitigate critical issues.
To help solve these challenges, Datadog CoTerm enables you to livestream, record, and log the contents of your terminal sessions, which you can then easily replay in a video-like player and query using full-text search.
In this post, we’ll walk through how Datadog CoTerm helps your teams to:
- Communicate progress around incidents without friction
- Maintain compliance when making changes to production systems
- Review full attack lifecycles, not just isolated fragments of evidence
Communicate progress transparently around incidents
Today, in order to share progress updates, findings, and learnings they have made in terminals, incident responders often copy and paste session information into general-purpose chat tools. This process tends to distort the formatting, wipe out important context through newer discussions, or force engineers to save sessions manually as scratch files—all of which still doesn’t make them easily accessible for others.
With CoTerm, incident responders can easily record their terminal in Datadog, either with a simple command at the beginning of their investigation or automatically by integrating CoTerm with their workflow when connecting to sensitive systems. Developers and other stakeholders can follow progress in Datadog as a livestream and learn from terminal activities of responders during the analysis and remediation without having to ask for manual updates. This helps to build confidence to make the right decisions during critical outages, as more eyes on the issue are better than siloed investigations. Developers on the sidelines of incidents can also make recommendations to help responders remediate sensitive issues without making costly mistakes.
Beyond this benefit of visibility into terminal sessions in the moment, having an easily accessible record of what incident responders have done is useful, for example to confirm what actions were taken or to train responders for similar future incidents. For accesibility, Datadog captures and stores the sessions and provides playback functionality based on the asciinema project. Datadog also extracts and stores any commands—with related context and attributes—as events, and the full terminal sessions as fully searchable logs, filtering out suspected secrets before they leave the local environment by using Datadog Sensitive Data Scanner and an entropy-based secrets detector. This enables the secure and transparent communication of progress and transfer of knowledge both during and after incidents.
Maintain compliance when making sensitive changes to production systems
Organizations often give their engineers, SREs, and other privileged individuals special access to critical production infrastructure. These organizations also need to comply with regulatory requirements that mandate detailed audit trails for user activities. Unfortunately, there hasn’t been any way for them to granularly monitor, record, and document the changes users make to these production systems in context.
Datadog’s CoTerm terminal recording provides a comprehensive way to log SSH sessions, including keystrokes and processes executed, without any manual overhead. As with incidents, terminal activities around sensitive system activities, including individual commands, can be made accessible later as logs, replays, and events in Datadog. On top of this, users can perform full-text search across all captured sessions.
To enable Datadog’s CoTerm terminal recording, you need to configure the SSH server to launch ddcoterm instead of the default shell when a user logs on. You can achieve this simply by configuring the sshd_config file and restarting the SSH daemon. After performing this step, users will be greeted with the appropriate details upon SSHing into the machine, as shown in the following recording.
Review full attack lifecycles, not just isolated fragments
Most security solutions give users just a possibility of retrieving isolated commands that were executed as part of an attack, and then, only if those commands were detected in the first place. This doesn’t provide the context of how an attacker got access in the first place, which activities they tried to pursue beforehand and afterwards, and how they were finally able to compromise the system.
Session recordings with CoTerm can fill in the gaps by providing security teams with the entire context of how attacks happen. To achieve this, configure CoTerm to log and record SSH sessions automatically on sensitive systems. This allows users access to these sessions either in real time or asynchronously through replays. You can also configure CoTerm to trigger alerts for predefined security signals like specific executed commands, enabling rapid response to potential threats. This proactive approach significantly reduces the window of opportunity for attackers to exploit vulnerabilities and provides the context to respond even more effectively. And even after attacks, the recorded terminal sessions provide invaluable data for forensic analysis. Security teams can review the exact sequence of commands executed by the attacker, understand their methods, and identify compromised systems.
Maintain a comprehensive view of terminal activity
CoTerm by Datadog enables you to monitor access to all your production systems, not just through logs, but through exact recordings of terminal access and activity. These recordings allow you to provide detailed logs to auditors, as well as perform detailed investigations of inappropriate access to your critical infrastructures and systems. Finally, having access to recordings of terminal activity gives engineering teams the significant benefit of helping them learn effectively from ongoing and former incidents.
Request access to the Product Preview of CoTerm by Datadog.
If you don’t already have a Datadog account, you can sign up for a 14-day free trial to get started.