Large organizations often rely on multiple monitoring tools, security platforms, and auditing systems to meet the diverse needs of their observability, security, engineering, and compliance teams. Because these teams may use the same logs for many different use cases—including detecting potential threats or breaches, troubleshooting errors, and gauging the effectiveness of new features—it can be difficult to effectively standardize and route data. Additionally, organizations can struggle with tool sprawl in the absence of a strong central observability team, particularly as they acquire new teams or migrate to the cloud.
Datadog Log Pipelines offers a fully managed, centralized hub for your logs that is easy to set up. You can ingest logs from your entire stack, parse and enrich them with contextual information, add tags for usage attribution, generate metrics, and quickly identify log anomalies. You can also use Sensitive Data Scanner, standard attributes, and granular tagging to enforce organization-wide conventions and industry regulations. Distributing these processed logs to users across your enterprise can be a challenge, however, as some teams may prefer to use platforms or environments outside Datadog to accommodate certain workflows.
We are excited to announce that Log Pipelines supports Log Forwarding to custom destinations, now generally available. Log Forwarding enables you to centralize log processing, enrichment, and routing so that you can easily send your logs from Datadog to Splunk, Elasticsearch, or HTTP endpoints. By leveraging rich filtering options and routing logs to multiple destinations, you can provide standardized logs to your teams and easily manage a wide variety of logging use cases. With Log Forwarding, you can:
- Centralize log processing while accommodating flexible workflows
- Enable compliance by duplicating logs across locations
Centralize log processing while accommodating flexible workflows
Datadog Log Pipelines allows you to ingest and transform your logs with features like grok parsing, remapping, and string extraction. Using Log Forwarding, you can take logs processed in Datadog pipelines and easily adapt them to the tools that work best for individual teams, with simple configuration and integration for your teams’ HTTP, Splunk, or Elasticsearch endpoints. This helps you centrally manage log processing, while still providing enough autonomy to your teams that they can efficiently analyze logs according to their specific requirements.
You can create custom destinations for external forwarding, then choose which logs you forward and how you forward them to fit your best practices. Log Forwarding provides you with role-based access control (RBAC) settings to manage who can create, edit or remove these destinations, helping you to fine-tune your security configuration and streamline audits. Additionally, Log Forwarding enables you to filter logs on an as-needed basis so that your teams receive only the data most relevant to them. This reduces the number of logs they need to store, cuts down on unnecessary noise, and helps prevent potential data leaks.
Let’s say your enterprise uses logs to monitor user authentication activity for troubleshooting system issues and detecting suspicious behavior. Your application and central observability teams have already adopted Datadog, but your security teams still follow established Security Information and Event Management (SIEM) workflows on a different observability tool. Using Log Forwarding, your central observability team can collect and process your logs in Log Pipelines, then easily forward the necessary logs to your security team’s external endpoint. This allows all of your enterprise users to work on their preferred platforms—your application teams can analyze their logs in Datadog using features like Application Performance Monitoring, and your security teams can still manage security threats using the application of their choice.
You can also use Log Forwarding to help your teams transition to new platforms. As you adopt Datadog Log Management, there may be certain teams that need to continue using existing solutions for contractual or business continuity reasons. You can easily ingest your logs in Datadog for centralized processing and parsing, then use Log Forwarding to route the logs to existing vendors. Each team can now migrate their workflows on their own schedule while still accessing standardized logs. Additionally, this dual shipping helps you ensure that downstream dependencies, such as data warehouses, continue to function as expected.
Enable compliance by duplicating logs across locations
Using Log Forwarding, you can send your logs to dedicated storage locations anywhere in the world. Geographically distributed organizations with many satellite offices—including government agencies, financial institutions, or insurance companies—often have to maintain copies of their logs for compliance or regulatory reasons. For example, some laws or standards stipulate that organizations must store their logs in a neutral location for a specified period of time or maintain local backups via their own file servers. Log Forwarding can help your teams access a consistent view of your logs no matter where they are located. This centralizes log pipelines, streamlines collaboration, and reduces mistakes that can result from users having outdated or inconsistent logs. Additionally, for longer-term storage, you can use Log Forwarding alongside Datadog’s existing archiving capabilities at no extra cost.
Shipping logs between locations also helps you facilitate projects with external partners or consultants. By providing an easy-to-use interface for building these integrations, Log Forwarding frees your teams from having to create custom tools for importing and exporting data. This allows them to focus on project tasks and deliver faster turnaround times.
Send processed logs to customized destinations with ease
With Datadog Log Forwarding, you get the best of both worlds—you can centrally process your logs according to industry regulations and your organization’s best practices while still giving your teams the autonomy they need to work effectively with their preferred tools. This flexibility enables you to easily support co-existence with other vendors, achieve compliance with industry regulations, ensure the accuracy of local backups, and streamline internal and external collaboration.
Log Forwarding is generally available to Datadog users. If you’re an existing customer, you can get started with Log Forwarding using our documentation. Or, if you’re not yet a Datadog customer, you can sign up for a 14-day free trial today.
