A primary goal for security teams is identifying specific threats to their environment, but they often face the daunting task of reviewing vast amounts of log data and alerts. Even with well-crafted detection rules, sifting through irrelevant data to pinpoint essential details for an investigation can be a significant challenge. This not only prolongs investigation times but also increases the risk of overlooking critical information.
To address these hurdles, Datadog Cloud SIEM enables security teams to add context to their detection rules and log searches with Datadog Reference Tables. This capability allows teams to easily filter out non-relevant data, keep investigations focused, and detect threats efficiently.
In this post, we’ll walk you through how reference tables allow you to:
- Use data that’s critical to your security teams
- Optimize your detection rules for fast, accurate signal generation
- Conduct efficient security investigations on historical logs
Add context to detection rules with reference tables
Datadog Reference Tables enable your security teams to supplement logs with custom metadata. Each table is built around a primary key—typically an ID present in your logs—paired with additional, contextual data linked to that key. For example, the risky_okta_users table below uses the usr_id field from Okta logs as the primary key and assigns a risk score to each user.
For your security teams, this capability allows them to bring rich and up-to-date security context to their detection rules, including:
- Data residing outside of standard logs, such as exposed keys and managed endpoints
- Third-party security feeds, including threat intelligence lists
- Sizeable datasets, like watchlists, known IP addresses, and indicators of compromise (IoCs)
Refresh tables with current data
As the threat landscape evolves, detection rules and their supporting datasets can quickly become outdated. This increases the risk of overlooking critical issues, which leaves your environment vulnerable to threats. With Datadog Reference Tables, your security teams can easily update and enrich the data used by detection rules, ensuring they continuously evaluate logs against the most up-to-date datasets, threat intelligence lists, and more.
Optimize your detection rules for fast, accurate signal generation
Writing effective detection rules is a constant challenge for security teams. While these rules are essential for identifying potential threats, they can often be too broad, generating an overwhelming number of alerts. This leads to wasted time as security teams sift through irrelevant notifications, which makes it difficult to focus on real threats. By incorporating custom data tables with detection rules, the investigative process becomes simpler and more efficient than traditional methods. Reference tables enable your security teams to fine-tune rules with specific criteria, quickly filter out noise, and minimize false positives.
To filter a detection rule using a reference table, navigate to the rule’s editor. Click the “Add” button next to the search query editor and select “Join with Reference Table.”
In the dropdown menu, choose the appropriate table and log field. Finally, select either the “IN” or “OUT” operator, depending on whether you want to include or exclude values from a specific table column.
Conduct in-depth security investigations on historical logs
Building secure applications requires deep visibility into network and service activity, with logs serving as a critical tool for monitoring. However, logs alone often lack the full context needed for security teams to investigate threats efficiently. To solve this problem, they need the ability to instantly query relevant logs and filter them using up-to-date threat intelligence.
With the Log Explorer, your security teams can filter logs at query time using their custom tables, ensuring they have accurate, up-to-date information for security investigations and critical audits. Datadog will then automatically enrich each log with that data, which gives your teams a more complete view of activity beyond what standard log data alone provides. The following example demonstrates how they can quickly sift through a large volume of Okta authentication logs to assess specific users with higher risk scores:
In addition to having the ability to analyze logs, Datadog allows your security teams to retain them for a standard 15 months or variably with Flex Logs. Flex Logs decouples the cost of log storage from the cost of querying, enabling you to keep your logs for the relative long term while still being able to instantly query them for audits and in-depth investigations.
Enhance your Cloud SIEM detection rules with Datadog Reference Tables
In this post, we explored how Datadog Reference Tables enhance your security team’s detection rules, ensuring they have the most up-to-date information to identify malicious activity and attacks. To dive deeper, check out our documentation on setting up reference tables. You can also explore our blog for practical examples of using reference tables with logs and building sufficient security coverage for your detection rules.
