Observability Pipelines | Datadog
Observability Pipelines

Observability Pipelines

Control log volumes, flexibly adopt your preferred security tools, and manage sensitive data at scale

Datadog Observability Pipelines empowers DevOps and Security teams to seamlessly aggregate and process logs within their own environment before routing them to cloud platforms, SIEM tools, data lakes, or other analytics solutions. By filtering, transforming logs to standards like OCSF, enriching logs with contextual data, and extracting metrics, you can optimize costs, flexibly adopt your preferred solutions, redact sensitive information, and more. With preconfigured pipeline templates and built-in performance monitors, Observability Pipelines makes it easy to design, deploy, and manage pipelines—all from a single, intuitive UI.

Cost-effective processing and routing of all your logs

  • Reduce log volumes by filtering logs before you route them, deduplicating logs, and dropping unnecessary fields
  • Generate metrics from your logs to reduce log volumes, track KPIs, gather insights, and more. Route metrics to Datadog to configure alerts, analyze trends and correlate them with other telemetry types
  • Stay within budget by imposing rule-based daily quotas or sampling strategies without losing access to mission-critical KPIs and trends
  • Save on egress costs by sending only valuable logs to your vendor and route noisy logs to long-term storage, such as AWS S3, Azure Blob Storage or Google Cloud
Cost-effective processing and routing of all your logs

Flexibly adopt your preferred security and observability solutions

  • Easily adopt your preferred SIEM, data lake, or logging solution at your own pace without sacrificing visibility by dual-shipping your logs to multiple destinations
  • Ensure seamless log migration from legacy, on-prem solutions to cloud environments with routing flexibility and budget control
  • Take advantage of best-of-breed solutions by classifying and routing logs based on use case, such as sending security logs to SIEM and DevOps logs to a log management platform
Simplify migrations

Transform and enrich logs to improve troubleshooting, governance, and analytics

  • Automatically apply 150+ out-of-the-box parsing rules or write custom rules to transform logs into an easily queryable, structured format using the Grok parser
  • Easily remap your logs to OCSF on the stream to standardize your security data before routing to your preferred security solutions or data lakes
  • Enhance security and improve data analysis using the GeoIP Parser, which helps you identify an IP address’s geographical location
  • Enrich logs with business-critical data, such as inserting a hostname within log messages using using custom tables, to enhance usability for downstream analysis, simplify debugging, increase searchability, and speed up Root Cause Analysis (RCA)
Meet your organization’s security and compliance requirements

Meet your organization’s security and compliance requirements

  • Redact or hash sensitive data such as credit cards, email addresses, IP addresses—before it leaves your environment using Datadog Sensitive Data Scanner
  • Comply with data residency laws and support Data Loss Prevention initiatives with full control over data routing, and enable built-in or user-defined rules to stay compliant with PCI, GDPR, HIPAA, CCPA, and more
  • Eliminate gaps in access control and ensure schema standardization by easily adding, copying, and dropping relevant attributes and tags
Enhance security insights and DevOps analytics

Build, monitor, and manage log pipelines through a single control plane

  • Easily get started with pipeline templates for common use cases such as dual shipping logs, reducing log volumes, and archiving data
  • Easily create, deploy, and manage your pipeline instances remotely using a point-and-click UI
  • Configure pipelines with confidence and diagnose production issues easily by viewing your data live as it flows through the pipeline using Live Capture
  • Monitor the health and performance of all pipelines deployed in your infrastructure in one centralized view, and get alerted on key issues
Build, monitor, and manage log pipelines through a single control plane

Observability Pipelines Product Brief

Control log volumes, flexibly adopt your preferred security tools, and manage sensitive data at scale.
Get the product brief

Resources

/blog/observability-pipelines/new-observability-pipelines-hero

BLOG

Aggregate, process, and route logs easily with Datadog Observability Pipelines
feature
observability pipelines
/blog/observability-pipelines-log-volume-control/hero-op-log-volumes

BLOG

Control your log volumes with Datadog Observability Pipelines
feature
observability pipelines
/blog/observability-pipelines-dual-ship-logs/dual-ship-logs-hero

BLOG

Dual ship logs with Datadog Observability Pipelines
feature
observability pipelines
/blog/observability-pipelines-sensitive-data-redaction/il-1243-sensitive-data-redaction-observability-pipelines-feature-announcement-240401-v1

BLOG

Redact sensitive data from your logs on-prem by using Observability Pipelines
feature
observability pipelines
View All Related Resources
Get started with Observability Pipelines today with a 14-day free-trial
view pricing