Datadog CoTerm: Never run the wrong terminal command again

For engineering teams investigating and resolving incidents together, it’s a common practice to use terminals alongside IDEs and applications like the AWS Console and Datadog. No matter how much tooling and automation a team builds, sometimes there’s just no substitute for dropping down to bash (or zsh, or fish). However, this power comes with peril—even the most careful engineers occasionally mistype commands, with serious consequences. Mistyped commands can lead to sticky situations such as programs executing in the wrong environment or infrastructure changes made without the correct approvals.

That’s why we’re excited to announce the Preview of CoTerm, Datadog’s solution for error-proof, team-powered command-line workflows. CoTerm provides real-time checks on sensitive commands directly in your terminal, enabling you to avoid missteps that can have major ramifications. You can try CoTerm today on macOS and Linux.

Check commands for safety

CoTerm can quickly and easily check for common errors in commands. Imagine you’re running kubectl to scale down a Kubernetes cluster used for development. You assume you’re in the development context and run kubectl scale. CoTerm intercepts the command, notices that you tried to run a risky operation without explicitly specifying --context, and tells you that you’re actually running the command against production:

CoTerm makes it easy to prevent mistakes like this, and it’s not limited to kubectl; you can write your own rules for any command.

Check commands for proper approvals

CoTerm can also be used for lightweight approval workflows, when commands are so sensitive that they need a second pair of eyes. Imagine that you are intentionally running kubectl in production because you are decommissioning a namespace that is no longer used. This is potentially dangerous—typing the wrong namespace could cause a major outage.

To avoid this, you configure CoTerm to require approval for kubectl delete or similarly sensitive operations in production. CoTerm intercepts the command, warns that it’s dangerous, and requires approval before execution:

CoTerm also automatically creates an approval request in Datadog Case Management:

In this scenario, kubectl delete will only run after it’s been approved by a coworker. Approvals can also be linked to incidents—command approval is a great way to lessen risk during high-stakes, time-sensitive incident remediation.

Record, browse, and replay terminal output in Datadog

CoTerm automatically records terminal output when a command is intercepted. In the previous examples, the full terminal session is browsable and replayable in Datadog. You can quickly and easily search through recordings to answer questions like, “Who ran this command in production recently?“

You can rest assured that terminal recordings won’t contain any PII or other sensitive information because CoTerm redacts sensitive data, such as keys and passwords, before sending the recordings to Datadog.

Recordings can also be useful for knowledge sharing with coworkers. Simply run ddcoterm and CoTerm will record your interactive shell session, which you or others can then watch in Datadog:

How CoTerm works

CoTerm shims commands by using the PATH environment variable. For example, let’s say you want to shim a dangerous command named foo. You would run ddcoterm shim create foo, which does the following:

1. Creates a lightweight script for launching CoTerm at ~/.ddcoterm/overrides/foo.
2. Adjusts PATH in your shell configuration so that ~/.ddcoterm/overrides/foo takes priority over the usual foo executable if necessary.

The next time you run foo, it will launch CoTerm, which quickly evaluates any user-defined rules for foo, generates warnings and approval requests if necessary, and then runs foo as usual.

CoTerm rules are Lua snippets embedded in CoTerm’s YAML configuration file, like this:

 
- command: "foo"
  rules:
    - rule: has_arg("bar")
       actions: ["approval", "record", "logs", "process_info"]

This rule tells CoTerm, “If foo is intercepted with a bar argument, require approval and record it.” This would apply to someone running foo bar but not foo baz.

Shimming commands is fast, and we aim for it to be unnoticeable. If the Lua rules for a command determine that an approval and a recording aren’t necessary, CoTerm can get out of the way in single-digit milliseconds.

Try CoTerm today

CoTerm is available in Preview today. For details about installation, see our CoTerm docs. To get started even faster, follow these steps:

1. Run brew install coterm on macOS or our install script on Linux.
2. Run ddcoterm init to log in, create some default rules, and shim kubectl.
3. Try running ddcoterm to record an interactive session, or kubectl test-coterm to test shimming and approvals.

If you’re new to Datadog, you can sign up for a 14-day free trial.