Julie Wang
Software Engineer, Intern
Garrison Stauffer
Senior Software Engineer
NetFlow is a popular network protocol for collecting metadata about traffic flows across your environment so that it can be exported for analysis and monitoring. One of the most common issues that users encounter is NetFlow duplication, which occurs when identical flow records from the same conversation are recorded from different sources. Flow duplication inflates traffic data, undermining capacity planning and making top-talker rankings unreliable.
For example, let’s say you are in charge of monitoring a multi-floor office network with switches on the first and second floors, and a firewall on the second floor that connects the office to the internet. If NetFlow is enabled for each device, a request from a host on the first floor out the firewall on the second gets exported three times, once by each device that saw it. When you look at analytics for that host, it will seem like it sent three times as many bytes as it actually did, hindering your ability to get accurate network performance metrics.
In this post, we will cover:
Why flows get duplicated
A NetFlow record represents a flow with a shared set of attributes: source IP, destination IP, source port, destination port, and protocol, along with some additional metadata. These records can be duplicated when different sources in your network report the same flow.
There are two causes of flow duplication:
Ingress/egress duplication: A device can be configured to export flow records for traffic entering an interface (ingress) and traffic leaving it (egress). If both ingress and egress interfaces on a device enable NetFlow monitoring, then packets will be double-counted per device. The fix for this is simple: Configure NetFlow monitoring on only egress interfaces for your devices.
Multi-exporter duplication: Traffic between two hosts rarely crosses only one device. It might pass through a switch, a router, and a firewall, and each hop can export its own record for the same conversation. In a three-device scenario, a 500 KB conversation between a host and an ISP can look like 1.5 MB. This gets more complicated on networks with redundant paths, like leaf-spine topologies. Traffic between two hosts that reroutes to a different path—for example, if a device goes down, or as a result of load balancing—can cause the same conversation to be re-reported across different paths.
Multi-exporter duplication is the more complex of the two causes of NetFlow duplication, so let’s look more closely at how to solve this issue.
How to address multi-exporter NetFlow duplication
Multi-exporter duplication is challenging because nothing is misconfigured. Monitoring multiple network devices is common and enables network engineers to investigate network issues more granularly, leading to better-informed monitoring and capacity planning. A conversation crossing multiple devices on its way from one host to another is not an issue to be remediated; it’s simply how most networks are built.
The practical approach to avoiding multi-exporter duplication is to select specific devices that will act as authoritative reporters for various modes of traffic in your network. This can be achieved in two ways:
1. Selectively enabling NetFlow on a few high-traffic devices: The idea here is to enable NetFlow on only the devices and interfaces that see the most traffic in your network. For a small network, a good choice would be to enable NetFlow on the central firewall or router linking your network to the rest of the internet. This approach is simple and involves straightforward device-level configuration changes. However, using this method runs the risk of losing information, as you no longer have insight into the flow of traffic through non-central devices.
2. Assigning specific device roles and explicitly filtering for them in dashboards: This approach involves assigning device roles in your network topology—for example, border for devices at the edge of the network that handle north/south traffic, and core for devices that move large numbers of east-west traffic. You can filter dashboards by these roles to exclude duplicate flows reported by other devices. In our example, filtering for border devices gives accurate counts for conversations between internal hosts and the broader internet because it filters out reporting from internal devices. The main drawback of this approach is the added complexity of assigning roles and querying for them, since flows will not be deduplicated out of the box (as they are with the previous approach).
How to choose your devices for deduplication
How can you choose what devices to act as authoritative sources of traffic flow data in your network? Let’s answer this question by using a realistic example. Suppose you are a network engineer in charge of gathering network traffic data for your company. Your company network is a WAN, with offices in New York City and Boston connected over a private tunnel. Both offices employ a leaf-spine pattern behind a central router.
Internet-bound traffic
Suppose an employee in the New York office wants to connect to their favorite music streaming service while they work. This action initiates a flow from the employee’s laptop through a leaf switch, through one of the two spine switches, and out through the NYC router. The leaf and spine devices are a poor choice as authoritative reporters here: Most ECMP routers hash per flow for load balancing, but some implementations allow a single flow to be load-balanced across multiple paths. As a result, tagging both spine switches the same way would over-report that one conversation.
If your network has ECMP or any other form of multi-path routing, choose a device further along the path as the authoritative source, such as one that every load-balanced path eventually feeds into. In our example, the NYC office router is a good choice. Filtering for your central router’s flows results in numbers that are accurately deduplicated for any traffic between the office and the internet.
Internal traffic
Now suppose the employee wants to talk to their colleague, who is also in the NYC office, over Slack. Their conversation never reaches the router, so filtering by the router device shows nothing. For internal traffic, selecting one of the two spine switches as the authoritative source is the better choice. Filtering for border devices gives accurate numbers for traffic entering and leaving the network; however, traffic between two internal hosts will not be seen by a border device.
This is a common scenario in data centers, where neighboring services constantly share data with each other. As such, gaining visibility into east-west traffic requires deeper inspection of relevant devices. We recommend filtering traffic for your network’s high-volume backbone devices, such as spine switches or load balancers, to include internal traffic measurements.
Cross-site traffic
Lastly, suppose the employee wants to reach a colleague in Boston over the private tunnel. The flow exits the NYC router and enters through the Boston router, both of which are tagged border. Filtering on border devices picks up the NYC-to-Boston conversation at both ends, resulting in double-counting.
We recommend filtering on a second dimension for additional scope, something like site:boston and site:new-york, to let a query isolate one site from the other. This helps filter against flows that can potentially be duplicated as the result of traveling through multiple border or core devices.
How to deduplicate NetFlow traffic in Datadog
Datadog NetFlow Monitoring, part of Network Device Monitoring, collects NetFlow records from your routers, switches, and firewalls, and aggregates them into dashboards that enable you to see traffic volume, conversations, and top talkers by protocol or port. Fortunately, deduplicating NetFlow records on the Datadog platform is straightforward. (Note that selective NetFlow monitoring is a device configuration-level change.)
Set device tags in the Datadog UI: Tag devices in Datadog by going to Network Devices → Devices → the specific device’s device page → Device Settings → Tags, where you can add a tag directly.
Filter for tags in NetFlow Monitoring: Head to the NetFlow page in Network Monitoring. Next to the search bar, click the magnifying glass. Then, type the key-value pair associated with your tag. For example, to search for border-classified devices under the device_role tag, query for device_role:border. The resulting dashboard will now show only flows tagged with devices that have those tags and will hide duplicate flows reported by other devices.
Get clear, accurate insight into NetFlow traffic data
Multi-exporter NetFlow duplication is a complex problem, and there is no single, one-size-fits-all solution. However, you can work around this issue by understanding your network topology and implementing the right filters on your traffic data.
To start collecting and deduplicating NetFlow data today, check out our NetFlow Monitoring documentation and configure your devices to collect NetFlow traffic. If you’re new to Datadog, sign up for a 14-day free trial.
