Every security engineer has experienced this issue: after spending a lot of time creating a new SIEM alert to catch malicious behavior, you deploy it, only to find there are over 100 service accounts triggering false positives. Your SIEM is suddenly flooded with false alerts, and your team is overwhelmed as a result. You then spend hours or even days investigating these alerts and fixing your detection rule, hoping it will work better next time.
Oftentimes, security teams grapple with the challenge of deploying new threat detections effectively. Without rigorously testing detection rules on historical logs, these mechanisms can trigger a flood of false positives, which overwhelmes analysts with disruptive “alert storms” and obscures genuine threats.
In software development, engineers create tests to ensure their code functions correctly. This practice helps prevent regressions and enhances reliability. Similarly, in the field of security, we can use these principles (with Detection-as-Code) to improve signal efficacy.
With Datadog Cloud SIEM, you can test your rules in four ways:
- Backtesting: Run your rule against a specified number of days of historical logs
- Unit testing: Validate your detection rules for logic and syntax
- Simulation: Run your rules against reproduced attacks
- Production: Analyze real data and send valuable signals to your team
In this post, we will cover how to backtest your newly built detection rules with Datadog Cloud SIEM Historical Jobs.
What are Historical Jobs?
With the availability of Cloud SIEM’s new historical engine, Historical Jobs enable you to run your detections against historical logs stored in Datadog. Historical jobs are one-time executable queries that analyze a specified period of historical data. Unlike real-time rules, these queries do not run continuously but are executed as needed to review data from previously generated logs. The results of these historical jobs are barebones versions of signals, providing essential insights into potential threats or anomalies identified within the associated logs. This approach allows security teams to conduct thorough investigations of past events, uncover activity patterns, and understand the context of previous security incidents.
How does backtesting work?
Historical Jobs run against historical logs no matter if they are stored in a standard index, flex, or rehydrated from archives. To get started, you can select a custom detection rule as a candidate for testing by clicking the three-dot menu and “Run as Historical Job.” In the configuration menu, select the storage layer where logs are present, specify the time range for the historical logs you want to include in the job, and then click “Run Historical Job” to create it.
Upon completion of a job, you can view it and all other available jobs in the Historical Jobs list to investigate the triggered results further.
It’s easy to view results for a specific job by selecting one from this list, which opens up the Job Side Panel.
These results will mimic the rule’s generated signals and give you further insight into:
- Total Detected Results: How many signals would have generated from backtesting
- Total Matched Logs: How many logs matched the detection in the given timeframe
After reviewing the results generated from backtesting, you can confidently promote a subset of them to signals if immediate action is needed. When converting to a signal, you can also manually set the signal’s level of severity and notification target.
Create valuable signals from logs with confidence
Datadog Cloud SIEM Historical Jobs allows security teams to better understand the effect of new or modified detections in their environment and build confidence that new rules will generate valuable signals at the right time and in the right manner.
In this post, we walked you through how to backtest new detection rules with Historical Jobs, but stay tuned for additional blogs in this series where we will also look at how to unit test and simulate attacks for your detection rules directly in Datadog Cloud SIEM. If you don’t already have a Datadog account, you can sign up for a 14-day free trial to get started.
