Organizations often struggle to maintain visibility and control over their distributed cloud infrastructure, where changes in a single resource can have cascading effects throughout the system and potentially cause disruptions. In these environments, infrastructure changes that lead to incidents are often hard to troubleshoot—especially when teams are using disparate tools with siloed data—leading to longer resolution times, more downtime, and negative business outcomes. To troubleshoot infrastructure issues effectively, teams need a centralized view where they can track configuration changes to their cloud resources.
Datadog’s Resource Catalog is a central hub for complete visibility into your cloud infrastructure, with a simple, intuitive interface where users can view, sort, and filter resources by category, cloud provider, region, environment, and more. Now, the Resource Catalog also includes the Recent Changes tab, available in Preview, where customers can track configuration changes across multi-cloud environments. These insights make it easier to diagnose infrastructure issues and help speed up your incident response workflows.
In this post, we’ll show you how to use the Recent Changes tab in the Datadog Resource Catalog to:
- Simplify and speed up troubleshooting of infrastructure issues
- Monitor configuration changes across your multi-cloud infrastructure
- Proactively alert on high-impact changes
Simplify and speed up troubleshooting of infrastructure issues
When developers and DevOps teams are paged about an active incident, one of the most common questions they ask is, “What has changed?” Answering this question is particularly difficult when determining what changes in infrastructure configuration occurred during the timeline of the incident. The Recent Changes tab solves this challenge by providing a consolidated view of the last week of changes in all resources listed in your Resource Catalog inventory, across all cloud providers (AWS, Azure, and Google Cloud).
For example, let’s say you are paged for an incident involving an elevated number of server errors for one of your production services. You have eliminated code changes to the service as well as dependency outages, and you now want to see any infrastructure configuration changes that happened around the time when the alert was first triggered. You open the Recent Changes tab, where you find a reverse chronological history of configuration changes, beginning with the most recent. Each of these change events is automatically grouped by resource ID, to consolidate updates for a single resource. From this view, you can use filters for cloud provider, environment, account, team, or service. So, even if you have thousands of cloud resources with changes, you can narrow down the results to see only recent configuration changes on resources related to the service you are troubleshooting.
On each row, you can click to open up a resource side panel that provides more detail into the nature of the change, including a side-by-side differential view of the resource’s configuration before and after the change. This enables you to see all properties that were updated and assess the likelihood that the service impact you’re troubleshooting was caused by this change.
For example, here you can see that an EBS volume was removed from an EC2 instance that powers the service you were troubleshooting, which may be the root cause of application errors.
In the resource sidepanel, you can also see the configuration history of the resource for the past week and view metrics, logs, traces, monitor statuses, and related resources.
After you’ve applied a remediation for the faulty configuration change, the Recent Changes section in the resource side panel update will update within seconds with the fix you just deployed. Now, you can easily verify the fix is correct by ensuring the resource’s metrics and monitor statuses are healthy—all without leaving this side panel.
Monitor configuration changes across your multi-cloud infrastructure
Even with a change management process in place that ensures all infrastructure configuration changes are made with code through your IaC provider, it’s important to validate that the changes to your production environment infrastructure adhere to organizational compliance policies—particularly in multi-cloud environments, where you may have best practices specific to AWS, Azure, and Google Cloud resources. As a DevOps admin, you can use Recent Changes to browse and analyze infrastructure changes across all of your cloud providers to ensure they are in line with best practices you’ve defined.
In addition to sorting recently changed infrastructure resources by cloud platform and resource type (e.g., hosts, databases), you can also filter down to the recently created or deleted resources with a single click on the query cards at the top of the page. These filters help you quickly to track infrastructure inventory changes week over week and scope them to the area of your multi-cloud environment that you’re most interested in.
The Resource Catalog also makes it easy for you to combine simple queries to further narrow down your investigation. For example, if you’re interested in tracking down increases in infrastructure and compute costs across your multi-cloud environment, you can see all the hosts that have been created over the past week that have not been deleted, as shown in the screenshot below.
Recent Changes also offers single-line summaries of the latest configuration change for each resource that has been updated in the last week, making it easy to scan for changes that need further inspection. In addition, you can use the Recent Changes search bar to search for changes across resource types. For example, you can search for changes made to EC2 Security Groups to identify updates that may be impacting network traffic to EC2 instances.
Proactively alert on high-impact infrastructure changes
Recent Changes is powered by change events sourced from your cloud providers. These events are enriched with useful tags collected from the cloud provider, such as the resource type, account, cluster, and more. You can use these events to create Event Monitors in Datadog that will proactively alert you on known risky changes.
Infrastructure issues can be challenging and time-consuming to root-cause—you often don’t know for certain what caused an issue until after you implement a remediation that fixes it. A proactive alert can help your team limit the severity of an incident by shortening the time to detecting risky changes and allow you to resolve issues quicker.
For example, let’s say you recently resolved an incident that was caused by an IAM role update on an EC2 instance. By creating a monitor that alerts you whenever updates are made to IAM roles on EC2 instances in your production environment, you can be notified when this type of change happens in the future, regardless of how the changes were deployed (via Terraform, or directly on the AWS console). This enables you to start investigating the issue early and make the necessary remediations before the change causes significant downstream impacts.
Get started monitoring cloud resource configuration changes
With high volumes of configuration changes occurring across your multi-cloud environment each day, it can be difficult to determine which updates may have impacted your service reliability. With the Recent Changes tab in the Datadog Resource Catalog, you can quickly view and filter infrastructure changes from your distributed cloud environment in a centralized location, helping you speed up your investigation and resolve issues faster.
To start troubleshooting with Recent Changes, simply sign up for the Preview and follow the instructions in our documentation to start sharing your cloud change events. If you’re not yet a Datadog user, you can sign up for a 14-day free trial today.
