Observability Pipelines

Datadog Observability Pipelines empowers your IT and Security teams to seamlessly aggregate, process, and route your logs in your own environment before directing to cloud environments, SIEM tools, data lakes, or other third-party analytics solutions. It helps you easily filter your logs on-premise, transform them to standards like OCSF, enrich them with contextual information, and extract metrics from your logs to help you control costs, boost analytics, simplify migrations, and safeguard sensitive data. With preconfigured pipeline templates and out-of-the-box performance monitors, you can easily design, deploy, manage, and monitor your pipelines from the Datadog UI.

Cost-effective processing and routing of all your logs

Reduce log volumes by filtering logs before you route them, deduplicating logs, and dropping unnecessary fields
Generate metrics from your logs to reduce log volumes, track KPIs, and gather insights, and more. Route metrics to Datadog to configure alerts, analyze trends and correlate them with other telemetry types
Stay within budget by imposing rule-based daily quotas or sampling strategies without losing access to mission-critical KPIs and trends
Save on egress costs by sending only valuable logs to your vendor and route noisy logs to long-term storage, such as AWS S3, Azure Blob Storage or Google Cloud

Simplify migrations and reduce vendor lock-in

Ensure seamless log migration from legacy, on-prem solutions to cloud environments with routing flexibility, and budget control
Adopt your preferred SIEM, data lake, and logging solutions at your pace own pace without sacrificing visibility with the ability to dual-ship your logs
Take advantage of best-of-breed solutions by classifying and routing logs based on use case, such as sending security logs to SIEM and DevOps logs to a log management platform

Meet your organization’s security and compliance requirements

Redact or hash sensitive data—such as credit cards, email addresses, IP addresses—before it leaves your environment using Datadog Sensitive Data Scanner
Comply with data residency laws and support Data Loss Prevention initiatives with full control over data routing, and enable built-in or user-defined rules to stay compliant with PCI, GDPR, HIPAA, CCPA, and more
Eliminate gaps in access control and ensure schema standardization by easily adding, copying, and dropping relevant attributes and tags

Enhance security insights and DevOps analytics

Easily remap your logs to OCSF on the stream to standardize your security data before routing to your preferred security solutions or data lakes
Automatically apply 150+ out-of-the-box parsing rules or write custom rules to transform logs into an easily queryable, structured format using the Grok parser
Enhance security and improve data analysis using the GeoIP Parser, which helps you identify an IP address’s geographical location
Enrich logs with business-critical data, such as inserting a hostname within log messages using using custom tables, to enhance usability for downstream analysis, simplify debugging, increase searchability, and speed up Root Cause Analysis (RCA)

Build, monitor, and manage log pipelines through a single control plane

Easily get started with pipeline templates for common use cases such as dual shipping logs, reducing log volume, and archiving data
Configure pipelines with confidence and diagnose production issues easily by viewing your data live as it flows through the pipeline using the Live Capture
Easily create, deploy, and manage remotely your pipeline instances using a point-and-click UI
Monitor the health and performance of all pipelines deployed in your infrastructure in one centralized view, and get alerted on key issues