Applications frequently need to provide authentication credentials to gain access to cloud services and other resources. However, these credentials present a security risk because they are notoriously difficult to keep out of code. According to a GitGuardian report, 10 million credentials were publicly committed to GitHub in 2022. Leaked credentials such as these are a major cause of data breaches and account takeovers.
Datadog has partnered with GitHub and GitGuardian to help protect our customers’ accounts from this risk. As part of this ongoing collaboration, we notify users as soon as we have detected that their Datadog API or application keys have been published in public repositories. In this post, we’ll explain how this process works and walk through strategies for reducing risk and mitigating the impact of leaked credentials.
How we identify leaked credentials
Datadog API and application keys enable our users to submit telemetry data to Datadog and send requests to the API. Most commonly, we see that keys are leaked in:
- Datadog Agent configuration files
- Documentation/READMEs for repositories
- In-line comments in code
- Scripts that use application keys and API keys to query or send data (e.g., as with the Datadog SDK or Terraform)
We joined forces with GitHub and GitGuardian to help protect our customers from attackers who may use these leaked credentials to gain unauthorized access to Datadog. To help GitHub identify potential Datadog keys, we provide a regular expression and associated detection rules. As GitHub scans the commits to public repositories in real time, it uses this information to detect potentially valid Datadog credentials before then notifying us about the findings. In addition, GitGuardian continuously scans through public repositories to find potential Datadog credentials and sends us all the relevant information if any are discovered.
We automatically verify this information from both sources as soon as we receive it. And whenever we do find a valid Datadog API or application key, we immediately notify the customer so that they can take action.
Email is one main way we notify users. We first attempt to email the creator of the key. If that user is no longer active, we email the admin users of that same Datadog organization. In the email, we provide information about the type of token that was leaked, who created it, and a URL of the page where it was detected. We also email any dedicated security contacts that have been configured for the Datadog account. As a rule of thumb, we recommend adding security contacts to ensure that the appropriate individuals get notified about leaked credentials and other important security-related events.
We also notify customers by posting an in-app notification on the API key management page and the application key management page.
Finally, users can also be notified about key leaks through Audit Trail: If you are using Audit Trail, you will see an Audit Trail event about the leaked token. In addition, you can also set up an Audit Trail monitor to track this event.
Mitigating the impact of leaked credentials
Once a key has been exposed publicly, it is virtually impossible to remove it from the internet. Even after you delete it from the public repository and scrub sensitive data from your commit history, you can’t do the same for forked and cloned versions of your repository. Beyond the risks of the leaked key persisting in these alternate locations, anyone who may have already gained access to it before it was deleted from the repository will still be able to use it. And since leaked credentials can be detected very quickly (e.g., GitGuardian is an external entity that can find leaks on GitHub in a matter of seconds), it’s possible for an attacker to take action very soon after the leak. This emphasizes the importance of taking immediate action to limit the impact of a leak.
Once a key is exposed, revoking it is the only guaranteed way to prevent attackers from using it to gain access to your account. To understand the potential impact of taking this step, you can check the key’s usage history on the API key management page of your Datadog account. If it has been used recently, you may have critical workflows that depend on the leaked key. Before revoking the key, you can create a new key and update your workflows—including but not limited to Agent configuration files—to utilize this new one instead. Once you are using the new key everywhere, you can then revoke the leaked key.
After getting notified about a leaked key, you should also review your account for suspicious activity, such as the adding or deleting of any users, or changes to role permissions. Also, Audit Trail records authentication events that provide visibility into the creation, modification, and deletion of API and application keys. You can search Audit Trail for the name or UUID of the leaked key to learn about any activity performed by this key, as well as who or what performed that activity.
Best practices for managing credentials
Although it’s essential to quickly revoke keys and monitor suspicious activity after a leak, it’s also important to prevent your credentials from getting leaked in the first place. One way to reduce the likelihood of a leak is to avoid including credentials in plaintext. Leaked keys are often found in the Agent configuration file (datadog.yaml), for example, where plaintext keys can be stored. One way you can avoid including API and app keys in datadog.yaml is by using the Datadog Agent’s secrets management package.
Besides keeping credentials out of code and configuration files, you can also implement a few best practices to start managing your Datadog credentials more safely:
- Define scopes for application keys
- Use service accounts to manage application keys
- Revoke unused API keys
- Create multiple API keys for different purposes
Define scopes for your application keys
Application keys should be assigned only the minimum level of permissions needed, adhering to the principle of least privilege. Following this principle reduces the likelihood that a leaked key will have elevated permissions, which in turn helps limit the potential impact of leaks.
Datadog application keys inherit the permissions of the users who have created them, but these permissions are often broader than what the application keys actually require. Specifying scopes for your application keys can help. Scopes restrict application keys to a narrow set of capabilities in Datadog, allowing you to grant each application key only the types of permissions it needs. For example, an application may need the permission to access dashboards, but it should not be able to take any other actions. In that case, you can specify a scope that grants read access to dashboards but provides no other permissions. Note that scopes are meant to limit the privileges assigned to an application key; they cannot grant any privileges that are not already available to the creator of the key.
Use service accounts to manage application keys
Similar to how Datadog application keys inherit the permissions of the users who created them, the validity of the keys also depends on the status of the users who created them. If a user’s account is deactivated (for example, after they resign from the company), any application keys they have created will automatically get revoked. Once the key is revoked, however, any scripts or applications that are currently using that key will break.
Service accounts are designed to help solve this problem. They are a good option for managing application keys that are intended to be used by different people because they are not tied to any individual user of your Datadog organization. By using service accounts, you know that any activity associated with that application key is tied to that service account, rather than a particular user account, which may not be specific enough.
Revoke unused API keys
Although Datadog will notify you about key leaks that GitHub or GitGuardian have identified on public GitHub repositories, keys may also get leaked in other places. With this in mind, we recommend that you review your key usage activity on a regular basis and revoke inactive keys. Each key represents a security risk because it could potentially be leaked, so revoking keys that aren’t being used helps reduce risk exposure and the number of keys whose activity you need to monitor.
On the API key management page of your Datadog account, you can see the last time an API key was used over the past 30 days (or 90 days, if you’re an Audit Trail customer). If it has not been used in that period, this column will say “No recorded usage,” indicating that you should consider revoking it. Note a crucial difference that is relevant here between API keys and application keys. If an employee leaves your company and their Datadog account is deactivated, their application key(s) will automatically be revoked. However, any API keys they have created will remain valid. For this reason, unused API keys present a needless risk to your organization that lasts until you revoke them.
Create multiple API keys for different purposes
Beyond deleting unused API keys, we also recommend creating multiple API keys—each used for a distinct and limited purpose—as opposed to using a single API key for all scenarios in your organization. Creating multiple API keys for different teams and purposes can be helpful for tracking usage in a more granular way and for limiting the potential impact of any particular leak. If you get notified that a key has been exposed, you will be able to quickly determine which applications are affected and limit the scope of impact by specifically focusing on replacing the key in those specific workflows.
Keys to safeguarding your account
At Datadog, we aim to protect our customers by making sure that no one can gain unauthorized access to their accounts. Notifying our users about publicly exposed API and application keys is just one example of how we are taking concrete steps toward that goal. Since we first started working with GitHub and GitGuardian on this initiative, Datadog has sent thousands of notifications to customers to help them quickly remediate the impact of leaked credentials. And looking forward, the Datadog Trust & Safety team will remain committed to working on this and other projects that help our customers safeguard their data.
