Mahashree Rajendran
Mohammad Jama
As organizations scale their cloud operations, object storage costs and performance issues often escalate. With petabytes of data, thousands of buckets, and millions of objects—especially from AI and data-intensive workloads—minor inefficiencies can quickly drive up expenses and degrade service quality.
Datadog Storage Monitoring helps you see exactly how your cloud storage is used, reduce unnecessary costs, optimize performance, and improve operational efficiency across Amazon S3, Google Cloud Storage, and Azure Blob Storage. Storage Monitoring now provides prefix-level metrics for Amazon S3 across three key use cases: cost optimization, performance optimization, and operational efficiency.
In this post, we'll explore how new metrics for Amazon S3 in Storage Monitoring help you:
- Implement and manage S3 lifecycle and retention policies
- Quickly identify opportunities to optimize S3 storage costs
- Identify high-traffic S3 buckets and resolve bottlenecks
Implement and manage S3 lifecycle and retention policies
Configurations like versioning in S3 buckets provides valuable protection against accidental deletions and overwrites. But without proper lifecycle management, version accumulation can lead to increased storage costs and management complexity. Storage Monitoring provides visibility into bucket-level lifecycle and retention policies directly within the Datadog platform, making it possible to understand existing configurations and identify opportunities for improvement—all in one place. This centralized view helps you enforce organization-wide storage policies without needing to switch between multiple AWS console pages.
For example, Storage Monitoring might identify a versioned bucket accumulating excessive non-current versions without appropriate lifecycle rules. The detailed metrics for this bucket would show the impact on storage consumption, along with current lifecycle policy settings.
With this visibility into your S3 bucket configurations, you can implement appropriate lifecycle rules to manage non-current versions, moving them to lower-cost storage tiers or expiring them after appropriate retention periods.
Quickly identify opportunities to reduce S3 storage costs
Organizations often store data in the S3 Standard storage class by default, but this tier isn't cost-effective for infrequently accessed data. Identifying which data can be moved to lower-cost storage tiers typically requires deep analysis of access patterns and object metadata—especially on a prefix level.
Storage Monitoring's new request metrics by prefix for Amazon S3 enable you to visualize exactly the prefixes that are actively used and, conversely, those that contain rarely accessed data. Combined with object count and age metrics, you can quickly identify opportunities to transition infrequently accessed data to lower-cost storage classes.
Let's say you have a workload that's fine-tuning an LLM model. Storage Monitoring can show that your training dataset prefixes contain terabytes of data that were heavily accessed during initial training but haven't been retrieved since the model was deployed five months ago. In the following example screenshot, a specific AI dataset prefix is selected, and the sidepanel shows that the objects in that prefix haven't been accessed in over a month.
These metrics enable you to identify exactly which data would benefit from transitioning to Glacier or Glacier Instant Retrieval and, as a result, implement appropriate lifecycle policies through the S3 console or AWS CLI.
Identify high-traffic S3 buckets and resolve bottlenecks
Performance issues such as throttling or latency are often caused by hot prefixes, which are paths that receive more requests than S3 can efficiently handle. But pinpointing these hotspots requires granular visibility that most teams don't have. Storage Monitoring's new latency by prefix metrics provide unprecedented visibility into exactly where performance bottlenecks are occurring.
For example, Storage Monitoring might show that a specific prefix in your media processing bucket is experiencing high latency during peak processing times. Using the Summary graph, you can identify high-traffic buckets and then zero in on the corresponding prefix. In the screenshot below, the treemap breaks down latency by prefix within the storage-monitoring-test-saleel bucket. The largest blocks represent prefixes with the highest average latency, such as tb and tb/firstlevel.
These detailed metrics can provide clues to help you implement design patterns that will better distribute the workload, such as using randomized prefixes to avoid throttling.
Get granular visibility into storage usage and patterns
Storage Monitoring helps you understand exactly how your S3 storage is being used with granular metrics that enable you to optimize costs, performance, and operational efficiency. Each metric provides clear visibility into your storage usage patterns and configurations, enabling you to make informed decisions.
Storage Monitoring for S3 is available today, along with object size metrics for Google Cloud Storage and Azure Blob Storage. Request-related metrics for Google Cloud Storage and Azure Blob Storage are coming soon. Storage Monitoring is currently in Preview; you can sign up here to receive updates and check out our documentation to learn more. And if you're not yet a Datadog user, you can get started with a 14-day free trial.
