With Amazon S3’s scalable object storage, you can store and manage billions of objects across multiple AWS accounts, regions, and storage classes. S3 Storage Lens provides 29 useful metrics that give you deeper visibility into your S3 usage and activity across your entire organization. We are proud to be a pre-integrated AWS partner using the new CloudWatch publishing option to bring S3 Storage Lens metrics into Datadog for enhanced S3 storage monitoring. In this post, we’ll show you how this integration can help you:
- Monitor S3 storage trends alongside your other infrastructure and applications
- Optimize S3 costs
- Ensure that your organization is following data protection best practices
Get more insight into S3 storage trends in any context
Datadog’s out-of-the-box dashboard—shown above—lets you easily visualize and explore all S3 Storage Lens metrics, including the number of objects and bytes you’ve stored in S3 over time. Other widgets on the dashboard help you monitor your S3 cost efficiency, data protection practices, request activity, and errors. You can use the dashboard’s template variables to easily filter your metrics by AWS organization, account, region, bucket, and storage class. This granular view into your S3 Storage Lens metrics can help you understand organization-wide changes and trends in your S3 usage, costs, and security.
And you can customize your dashboard to add context to your S3 Storage Lens metrics. Once you’ve cloned the dashboard, you can add widgets to track the performance of your infrastructure and other AWS services alongside your S3 Storage Lens metrics.
Optimize S3 costs across all of your accounts
S3 Storage Lens metrics provide information about non-current object versions and delete markers, as shown in the screenshot below. These metrics represent opportunities to reduce your S3 storage costs by deleting unused objects. You’ll also see information about incomplete multipart uploads, which can produce unusable data that remains in S3 until you explicitly delete it, for example by using an S3 Lifecycle rule.
Datadog’s S3 Storage Lens integration also helps you understand how your organization accesses data in S3, which can be useful for further optimizing costs. AWS charges for data transfer and retrieval per request and per byte, and the costs vary across storage classes. Some retrieval methods (such as the GetObject SDK action) return an entire object, but you may be able to reduce costs by using Amazon S3 Select to query the object using SQL, allowing you to retrieve only the data you need.
The screenshot below shows an excerpt of the out-of-the-box dashboard that can help you understand your organization’s costs. For example, you can view a toplist of buckets that have the lowest data retrieval rates (indicating that they store infrequently used data), and consider moving them to a lower-cost storage class. You can also see the number of bytes scanned by S3 Select—indicating the amount of data processed by queries—and the number of bytes returned—indicating the aggregate size of all result sets. The buckets with the most bytes scanned might represent an opportunity to improve the performance and reduce the cost of your S3 Select queries by using the ScanRange parameter to scan only a subset of an object’s data.
Ensure that your S3 data is protected
AWS best practices prescribe a number of steps you should take to protect your S3 data against risks, including loss, corruption, and unauthorized changes. S3 guarantees durability to guard against data loss, but you still need to configure each bucket properly to take advantage of S3’s features for replication, versioning, and object locks.
The S3 Storage Lens dashboard provides a quick summary of the percentage of total storage that is encrypted, replicated, and protected with an object lock, as shown in the screenshot below. You can also monitor these metrics over time and even create alerts to notify you automatically if your data protection metrics drop below a threshold you specify, allowing you to quickly identify and remediate any incorrectly configured buckets or objects. For example, if you see a steady decrease in the percentage of objects with an object lock enabled, it could indicate that an application became misconfigured and began to create unprotected S3 objects. The toplists at the bottom show the buckets with the greatest risk of data loss, corruption, and exposure, which you can use to prioritize improvements to your S3 data protection.
And for continuous visibility into the protection and security of your S3 data, you can use Datadog Cloud SIEM’s out-of-the-box rules to monitor S3 permissions automatically.
Get started with S3 Storage Lens monitoring in Datadog
S3 Storage Lens complements our existing S3 integration to provide even deeper visibility into S3. To get started, configure S3 Storage Lens to publish metrics to CloudWatch, then enable the S3 Storage Lens integration. If you’re not already using Datadog, start today with a 14-day free trial.
