Jinwu Liu
As your applications scale and generate more telemetry, it becomes increasingly difficult to sift through the data and analyze it against cost, business functions, and security measures. Logs, events, and other telemetry on their own may not include enough meaningful context or readable details, leading to slower troubleshooting, inefficient business processes, and higher costs.
With Reference Tables (now generally available), you can import business-critical information into Datadog to allow you to enrich and join your logs, events, and more for deeper data correlations. You can define new entities like customer details, service names and information, and IP addresses by manually uploading a table of information via CSV file, cloud storage (e.g., S3, Google Cloud, Azure), or SaaS integrations (e.g., ServiceNow, Snowflake) automatically. The entities are represented by a primary key in a Reference Table and the associated metadata, which you can then use across a variety of business, application, and security use cases.
In this post, we’ll cover how you can use Reference Tables in Datadog to:
- Enrich your logs for fast investigations and analysis
- Enhance application security with custom threat intelligence
- Segment users for deeper product analytics
- Optimize cloud costs with Tag Pipelines
- Perform advanced querying with data transformation and aggregation tools
Enrich your logs for fast investigations and analysis
More logs come in as your application scales, which can mean your business-critical information is constantly changing from the time a log is ingested to when it is actually viewed. With Reference Tables, you can correlate and enrich your logs with dynamic metadata at ingestion time or query time to ease investigations and on-the-fly analysis. Adding up-to-date context enables intuitive visualization and logical groupings based on security or business use cases.
For example, a security analyst can use a Reference Table to dynamically flag suspicious activity by enriching logs with external threat intelligence. The analyst can use a Reference Table of malicious IPs that have been flagged for malware or phishing, removing the need for manual cross-referencing across disparate tools and speeding up further security investigations.
In addition to threat intelligence, you can enrich logs to help you troubleshoot application performance issues (e.g., mapping error codes to human-readable descriptions) and organize logs by logical units like teams and cost centers for better visibility. Check out our dedicated blog on log enrichment and organization to learn more.
Enhance application security with custom threat intelligence
With Datadog Application Security Management (ASM), you can now upload Reference Tables containing threat intelligence indicators of compromise (IOCs) to enrich security traces. This “bring your own threat intelligence” model adds organizational context to your investigations and alerts. Once a table is created, threat intelligence enrichments become available in ASM traces, allowing security teams to analyze, filter, and investigate incidents more effectively. By joining traces with a Reference Table, you can correlate historical attack data, assess the impact of threats, and enhance detection capabilities.
Segment users for deeper product analytics
Manually tagging or grouping large numbers of customers into segments can be cumbersome. When dealing with user, product, and performance data en masse, Reference Tables allows you to create a new user segment directly from an already existing Reference Table. This enables teams to visualize and tackle user-specific requirements catered to different sets of data, while automatically syncing future updates.
For example, let’s assume you’re a SaaS company and have free users, premium users, and enterprise users. Uploading a Reference Table of your users and what type of users they are enables you to observe user behavior and address targeted problems for each specific demographic. This table will then regularly update as new users are created.
Optimize cloud costs with Tag Pipelines
To monitor cloud and SaaS costs effectively, you need clear insight into how services, teams, and products drive spending. In Datadog Cloud Cost Management, you can use Reference Tables in Tag Pipelines to improve tagging on your cost data, helping you achieve precise cost attribution and reporting on any service, team, and product.
Tag Pipelines standardize tags across resources, ensuring consistent and accurate cost attribution for your entire organization. You can set rules to fix existing tags or add new tags for consistent and accurate tracking of your cloud and SaaS costs. With Reference Tables, you can add multiple tags at once; simply map values from your table’s primary key to cost tags, and Tag Pipelines will apply the relevant tags to your data.
For example, if you want to add information about which VPs, organizations, and business units different AWS and Azure accounts fall under, you can create a table and easily map the tags.
Perform advanced querying with data transformation and aggregation tools
Datadog has additional tools that let you do advanced data transformations and aggregations by joining external data from Reference Tables. These include Sheets, which is a spreadsheet tool that you can populate with Datadog data, enabling you to perform complex analysis and build reports without requiring technical expertise. The DDSQL Editor is another tool that lets you get deeper visibility into your infrastructure by querying your resources with natural language or with DDSQL. And finally, Log Workspaces lets you perform complex queries, such as combining attributes from multiple log sources or transforming log data to analyze your logs.
Boost observability using critical business information with Reference Tables
Reference Tables allows you to enrich and join your telemetry to provide valuable context for richer categorization, analysis, and security. By creating more relevant data, teams can improve troublehshooting, bolster security, optimize costs, and much more.
Check out our documentation to learn more about Reference Tables and how you can use them to keep track of your infrastructure. If you’re new to Datadog, you can get started today with a 14-day free trial.
