Amazon Virtual Private Cloud (Amazon VPC) is an isolated and secure virtual network in which you can deploy resources, such as Amazon Elastic Compute Cloud (EC2) and Amazon Relational Database Service (RDS) instances, while restricting their exposure to the internet. As part of your monitoring strategy, you can collect and analyze VPC flow logs, which record network traffic flow between VPC components. This information can help you better understand traffic patterns, optimize your security posture, and troubleshoot connectivity and configuration issues.
Amazon VPC already integrates with Amazon CloudWatch Logs and Amazon S3—and earlier today, AWS announced a new native integration with Amazon Data Firehose. We are proud to collaborate with AWS on this launch to enable our customers to easily send VPC flow logs from Amazon Data Firehose to Datadog for comprehensive monitoring and analysis. In this post, we’ll briefly discuss some benefits of sending VPC flow logs directly to Data Firehose, and also show you how to leverage Datadog Log Management for real-time log analysis.
VPC flow logs + Data Firehose
Amazon Data Firehose is a fully managed service for ingesting, transforming, and streaming real-time data to data stores and analytics platforms. Amazon VPC Flow Log’s out-of-the-box support for Amazon Data Firehose as a log destination means you no longer need to use custom tooling to ingest VPC flow logs from Amazon CloudWatch Logs or Amazon S3, which reduces your operational overhead and costs. Additionally, you can leverage Amazon Data Firehose’s powerful data transformation capabilities to process, enrich, and aggregate your logs before delivering them to your destinations of choice—all without having to maintain your own pipelines. You can even stream flow logs from multiple different source accounts to a single Firehose instance, enabling you to centralize your log collection and delivery.
To send VPC flow logs to Datadog, you will first need to create a Data Firehose delivery stream with Datadog as the destination. Then, create a VPC flow log with the destination as the delivery stream you have just created. The example below shows you how to accomplish this through the command line interface, but you can also use the Amazon EC2 and VPC consoles, as documented here.
aws ec2 create-flow-logs \
--resource-type VPC \
--resource-ids vpc-20e1195d \
--log-destination-type kinesis-data-firehose \
--traffic-type ALL \
--log-destination arn:aws:firehose:us-east1:059558860956:deliverystream/test-firehose \
--max-aggregation-interval 60 Search all of your VPC flow logs in real time
After you’ve completed the steps described above, you will begin to see VPC flow logs streaming into the Datadog Log Explorer. Datadog’s built-in log processing pipeline automatically parses your VPC flow logs and enriches them with key attributes, such as IP address, interface ID, and action (e.g., accept, reject). You can then easily search, filter, and group your logs by any of these attributes. Additionally, with Live Tail, you can view these logs in near real time, regardless of whether you’ve indexed them, which is particularly useful when you’re investigating a security issue.
Perform analytics on your VPC flow log data to uncover trends
Heavy traffic within your VPC can result in massive volumes of flow logs, which makes it increasingly difficult to spot potential issues. In Datadog, you can perform analytics on your high-cardinality log data to visualize trends over the long term and surface anomalies. The example graph below compares the number of log records that captured rejected traffic to the total number of logs.
To dig deeper, you can create a top list of client IP addresses from which the rejected traffic originates. Knowing which IP addresses are suspicious (such as 144.91.87.106 in this example) allows you to take the necessary steps to protect your environment.
You can also add these visualizations to any of your dashboards to view them side-by-side with metrics, traces, and other types of monitoring data from the rest of your stack.
Start monitoring VPC flow logs with Datadog Log Management today
Amazon VPC Flow Log’s native integration with Amazon Data Firehose streamlines the processing and delivery of VPC flow logs. And with our Amazon Data Firehose integration, you can seamlessly and reliably stream your logs to Datadog for monitoring, correlation, and analysis alongside the rest of your telemetry data. Check out our documentation to get started with VPC flow logging.
New to Datadog? Sign up for a 14-day free trial today.
