Data Streams Monitoring

Datadog Data Streams Monitoring (DSM) enables you to easily track and improve the performance of event-driven applications that use Kafka and RabbitMQ. DSM automatically maps dependencies between all services and queues and provides out-of-the-box metrics that help you measure end-to-end latencies, locate faulty queues or services and their respective owners, and remediate floods of backed-up messages so that you can avoid critical downtime and meet strict SLAs.

Detect slowdowns anywhere in your event-driven applications

Measure latency between any two points throughout your pipeline, including queues, producers, and consumers
Easily identify delayed messages by viewing actionable lag metrics both in seconds and offset
Quickly detect abnormal latency or throughput across your streaming data pipelines by creating threshold, anomaly, and outlier monitors

Localize and resolve pipeline bottlenecks and failures

Easily detect failing dependencies between queues, producers, and consumers with automatic, end-to-end pipeline topology mapping
Triage whether delays are originating from issues with application services or queueing systems, and identify the owning team
Prioritize troubleshooting efforts by impact of delay, with message backup volumes for services and queues

Remediate floods of backed-up messages

Quickly detect and resolve downstream bottlenecks to prevent long lags, message downtime, and delayed messages
Spot where bursts in message flow may be occurring upstream with automated consumer lag notifications for every service
Visually track the flow of messages across all dependent services and queues to ensure you constantly meet your SLAs

Correlate streaming data pipeline metrics with all telemetry in context

Resolve application issues faster by seamlessly pivoting between DSM metrics, distributed traces, infrastructure metrics, and logs
Gain complete pipeline visibility by adding latency and throughput metrics to your Kafka or RabbitMQ integration dashboards
Streamline incident response with quick access to service owners, on-call engineers, and critical resources via Service Catalog

Correlate streaming data pipeline metrics with other telemetry in context

Building streaming data pipelines to power our digital trading platform requires visibility into throughput and latency across the entire system, including all queues and services. With Data Streams Monitoring our team is able to proactively discover performance bottlenecks and optimize stream processing to ensure maximum throughput and low latency delivery of data to our customers.

Darren Furr Solutions Architect at MarketAxess

Datadog Data Streams Monitoring allows us to easily map the topology of our Kafka pipelines across our globally-spread clusters, monitor end-to-end latency, and identify sources of incidents across our entire pipeline in one place. Having DSM integrate seamlessly with APM helps us ensure that millions of gamers around the world always get the best possible experience.