Leverage Cloudflare logs for cost optimization, troubleshooting, and security

Cloudflare is a content delivery network (CDN) that helps businesses accelerate, protect, and optimize their websites, applications, and APIs. It acts as a reverse proxy, sitting between users and a website’s origin server to provide DDoS protection, web application firewall (WAF), CDN caching, and load balancing. Companies use Cloudflare to improve site performance, reduce bandwidth costs, enhance security against cyber threats, and ensure high availability through Cloudflare’s distributed network of data centers worldwide.

Cloudflare emits a number of different types of logs, including HTTP request logs, firewall event logs, access logs, DNS query logs, load balancer logs, and more. Each of these log types contain a wealth of information that can give you insight into the health and performance of your system. However, this abundance of data can make it difficult to find the signal in the noise. With the right knowledge, Cloudflare logs can be utilized for a number of different purposes.

In this post, we’ll explain the anatomy of a Cloudflare log, and show you how to utilize the different fields for:

• Debugging and troubleshooting
• Managing cost
• Security monitoring
• Compliance and auditing Finally, we’ll discuss how to monitor Cloudflare logs with Datadog.

The anatomy of a Cloudflare log

Cloudflare logs provide detailed insights into HTTP requests, including origin and response metadata, as well as security, TLS, and encryption information. Logs are structured as JSON objects, with each entry representing a single HTTP request processed by Cloudflare.

Here are some key fields to take note of in a Cloudflare log:

How to use Cloudflare logs for debugging and troubleshooting

Because Cloudflare logs contain a wealth of information, it’s important to know what you’re looking for so you can quickly and easily extract the relevant data when you’re troubleshooting an issue and need to surface relevant information fast. Let’s look at how you can use Cloudflare logs to resolve an incident quickly if you know what you’re looking for.

EdgeResponseStatus represents the HTTP response code (e.g., 404, 500, 502, etc.) that Cloudflare returned to the client after processing the request. By analyzing this field, you can differentiate between origin server issues and Cloudflare-related errors, as well as detect routing, caching, or security rule misconfigurations.

For example, let’s say you have a stock trading app, and customers are reporting 522 Connection Timeout errors when trying to check prices. The price-check service is on an API hosted by Cloudflare. You want to analyze how often these errors occur and which origin servers are affected, so you surface these error logs to see which users (ClientIP) are affected most and which origin servers (OriginIP) fail most often. You want to know if the issue is with Cloudflare or your API, so you filter by EdgeResponseStatus, but find that the value in these log fields is 200, 201, or 403 (forbidden), meaning that when possible, Cloudflare sent the response in a timely manner.

You suspect the issue is with your API, so you investigate OriginResponseDurationMs and find elevated values across the logs (up to 20,000 from 5,000 ms). With this knowledge in hand, you contact the developer team that owns the price-check service and let them know that there may be an issue with the code.

How to use Cloudflare logs to manage cost

Cloudflare logs provide detailed traffic insights, which can help teams optimize usage and reduce unnecessary costs related to bandwidth, compute, and log storage. To help find the signal in the noise, let’s take a look at specific fields and how they can help make smarter decisions.

To continue our example from above, your trading app processes a high volume of traffic. You’re looking for ways to reduce your cloud spend and know that your provider charges high egress fees. You filter your Cloudflare logs for high EdgeResponseBytes, which shows the size of egress traffic sent to Cloudflare, then surface large requests.

You can then find the OriginIPs emitting the largest amount of traffic and route this traffic to Cloudflare R2, which enables you to leverage a lower-cost hosting option using Cloudflare’s Cloud Connector. Alternatively, you might consider implementing CDN compression to reduce transfer costs.

Another source of high egress costs can be a high number of requests that miss the cache. You can use Cloudflare logs to surface issues of this kind. When a request results in a CacheStatus: MISS, it means Cloudflare did not serve the content from cache and instead had to fetch it from the origin server, increasing traffic costs. If you notice this happening, you can enable Cache Everything for static assets (e.g., CSS, JavaScript, and images) that don’t change often and thus can be loaded directly from the cache instead of making repeated requests to the server. Alternatively, you could set long Cache-Control headers—for example, max-age=31536000 would set the cache lifetime to one year. Or, you can use Tiered Caching to minimize redundant origin fetches for OriginIPs that frequently generate CacheStatus: MISS logs.

How to use Cloudflare logs for security monitoring and threat detection

Cloudflare logs provide real-time insights into security events, helping teams detect threats, bot attacks, DDoS attempts, and suspicious activity. By analyzing log fields like ThreatScore, EdgeResponseStatus, ASN, and BotScore, you can proactively monitor, investigate, and mitigate security risks.

A regular best practice for DevSecOps engineers is to alert on requests with high ThreatScores and flag them for further investigation. You can identify OriginIPs that requests with high ThreatScores come from and see if they match up with known malicious OriginIPs from widely available sources like Project Honeypot.

Cloudflare logs can also reveal unusual request patterns, which can tip you off to attackers exploiting vulnerabilities in your system. For example, excessive POST requests may indicate an attempted brute-force attack or a DDoS attack. While a spike in traffic can often be a good thing—for example, if it is the result of marketing campaign success—you can look at other factors in your logs to confirm that the activity is suspicious, such as if the traffic is coming from a single OriginIP or ASN.

Bots are another issue to be wary of, as they can scrape sensitive data, enact brute-force logins, or overload APIs. You can surface logs with high values in the BotScore field to detect bot traffic. To combat bots, you can enable Cloudflare Bot Management to block bad bots. Alternatively, you can use WAF Custom Rules to filter traffic from suspicious ASNs. You can then track WAFAction values (e.g., ALLOW, BLOCK, CHALLENGE) to identify and mitigate attack patterns, fine-tune security rules, and reduce false positives.

How to use Cloudflare logs for compliance and auditing

Cloudflare logs help organizations meet compliance requirements for regulations like General Data Protection Regulation (GDPR), Health Insurance Portability and Accountability Act (HIPAA), Payment Card Industry Data Security Standard (PCI-DSS), and System and Organization Controls (SOC 2). Auditors require detailed event logs to review access control, policy enforcement, and incident response.

It’s important to keep the following types of files in long-term storage, available for compliance investigations and audits.

• HTTP request logs are critical for compliance, audits, and security investigations. These logs provide a detailed record of web traffic, helping organizations meet legal, regulatory, and operational requirements.

• Firewall/WAF logs track blocked and allowed traffic, helping organizations detect unauthorized access, policy violations, and cyber threats while ensuring compliance with industry standards. Many compliance frameworks require these log events for auditing purposes. For example, a PCI-DSS audit requires proof that firewall rules blocked unauthorized access attempts to payment APIs.

• Access and authentication logs track user logins, failed attempts, and access to sensitive resources. These logs are critical for ensuring regulatory compliance, security monitoring, and forensic investigations. For example, a HIPAA audit uses logs to check who accessed patient data and when. And a PCI-DSS audit requires login and access logs to verify that only authorized users accessed payment systems.

Maintaining these types of logs long-term can add to storage cost, so it’s useful to adopt a flexible retention strategy that can help mitigate operational overhead. In the next section, we’ll discuss how you can use Datadog to accomplish this.

How to monitor Cloudflare logs with Datadog

While Cloudflare logs offer a wealth of information, it can be difficult to sift through the high volumes that a system generates, particularly at scale. That’s why it’s important to adopt a log management solution that can help you surface what you need.

Datadog’s Cloudflare integration enables you to stream Cloudflare logs into Datadog Log Management and offers out-of-the-box dashboards for metric and log monitoring.

Users can also stream Cloudflare logs into Datadog Cloud SIEM to stay ahead of security threats. For example, say a security analyst monitoring Cloudflare logs in Datadog notices a spike in 403 Forbidden responses originating from a single IP across multiple endpoints, indicating a possible credential-stuffing attack. Pivoting to Cloud SIEM, they investigate further by correlating this IP with failed login attempts in authentication logs. After confirming the IP is part of a known botnet, they create a WAF rule in Cloudflare to block the malicious IP range, while also setting up SIEM alerts for future suspicious login activity.

Additionally, Datadog provides tooling for users to monitor Cloudflare load balancers, helping ensure dependable traffic throughput. For example, say a DevOps engineer for an e-commerce company monitors Cloudflare load balancer metrics in Datadog to ensure optimal traffic distribution across multiple origin servers. When a service starts experiencing increased response times, the engineer checks the Cloudflare dashboard to analyze traffic patterns, finding elevated cloudflare.load_balancer.latency and cloudflare.load_balancer.pool.round_trip_time.average. They pivot to Cloudflare logs to check for timeouts or origin health check failures. They find neither of these issues, simply a spike in traffic, likely due to a recent sale. To resolve the issue, they adjust load balancing weights to maintain high availability and prevent downtime.

Because Cloudflare can generate a high volume of logs, it’s important to adopt a solution that can retain the logs you need, while still keeping costs down. Datadog Flex Logs offers flexible retention policies for your log storage, so you can selectively keep logs queryable if you may need them at a moment’s notice (e.g., database query logs, API request logs, web server logs), while routing logs like the ones mentioned above to long-term archiving (e.g., Amazon S3, Cloudflare R2) for potential audits in the future. Additionally, Datadog Observability Pipelines can help you meet compliance requirements by redacting sensitive data from your on premise logging solution.

Monitor Cloudflare logs in a unified platform

In this post, we’ve highlighted the anatomy of a Cloudflare log and shown you how to get the most out of the information therein. We’ve also shown you how storing and monitoring your Cloudflare logs with Datadog helps you extract the most value from them.

If you’d like to learn more, check out our dedicated blog post, documentation, or, if you’re new to Datadog, sign up for a free trial to get started.