Logs provide invaluable information about issues you need to troubleshoot. In some circumstances, that may mean that you have to look back at old logs. For example, you may be running a security audit and need to analyze months-old HTTP request logs for a list of specific IP addresses over a period of time. Or you might need to investigate why a scheduled service never occurred, or run an exhaustive postmortem on incidents that happened over a couple months but that you suspect are related. In each scenario, you will likely need to look at logs that have been moved to long-term cold storage, where it’s difficult to access and use them for troubleshooting. But retrieving and querying these old logs can be slow, expensive, and work intensive.
Challenges retrieving archived logs
Whether you use a log management service or rely on a self-hosted tool, managing old logs comes with a myriad of challenges. Typically, logs archived in storage-optimized environments are difficult to access and search, as re-ingesting logs from several months (or years) ago involves querying very large volumes of raw and unindexed data. For large organizations, that could even mean waiting several days to search through applicable archives, which is more time lost that could be used for root cause analysis.
Another difficulty is that many log management solutions are unable to store logs at a sufficient level of granularity. This means additional expense because, instead of resurfacing only the relevant subset of archived log events, organizations have to move every single archived log within a set time range into costly hot storage environments in order to properly index and search them.
Searching and retrieving archived logs can also incur significant workload costs. Organizations that don’t already have a log management solution may have to spend extra development days building out tools that enable internal teams to query archived logs. This includes additional organizational overhead as you must extend and manage access to log storage across all the teams that need it.
Datadog’s Log Rehydration™
With Datadog’s Log Rehydration™—part of our Logging without Limits™ feature set—you can efficiently archive all of your logs and retrieve the exact ones you need, when you need them. Log Rehydration™ lets you quickly and easily pull archived logs back into your Datadog account so you can analyze and investigate old events. Read on to learn how you can get the most out of Log Rehydration™.
Bring past logs into the present
In order to use Log Rehydration™, first follow the steps to create an AWS S3 bucket and configure your Datadog account to archive your logs in it (in zipped JSON format). Once archived, you can quickly rehydrate logs and bring them into the Log Explorer for analysis at any time. Simply navigate to the Rehydrate from Archives tab of your account’s Logs Configuration page and click New Historical View.
To create a Historical View, select the archive where the logs you need are stored, specify the time period when those logs were produced, and select a retention period. Then you can define a query using the service you are investigating and any additional attributes (e.g.,status, @http.url_details.path, @customer_id, etc.) or free-text search terms you want to look for. For example, the following Historical View will scan a four-day period in our Prod Archive and rehydrate error logs from our web-store service that include the phrase “payment rejected.”
Once you’ve created a Historical View, Datadog will scan the S3 archive you selected and retrieve the logs that match the given criteria back into your account so you can perform your analysis.
Rehydrate with precision
Datadog’s Log Rehydration™ is fast, with the ability to scan and reindex terabytes of archived logs within hours. This means you can quickly access the information you need. You can make the process even more efficient by narrowing your search parameters. The more specific your search terms, the fewer log events Datadog will need to rehydrate and the faster you will receive results. When you query an archive you want to rehydrate from, Datadog lets you specify a time range down to the minute to limit the number of logs Datadog needs to search through. To narrow the rehydrated log results even further, you can also define a query with precise search terms or attributes, so no time is wasted collecting superfluous logs you have no use for.
This means that there is no long delay to re-ingest the logs you need. You can begin using your old logs immediately for troubleshooting and root cause analysis.
Organize old logs with archives
Some organizations may have extremely large log archives, or may need to rehydrate certain subsets of log data more frequently. In these cases you can create separate archives for logs that you need to rehydrate more often to keep rehydration efficient. This will help ensure your old logs are well organized so that, when you need to rehydrate them, you’ll know exactly where to look. For instance, if your organization produces a lot of HTTP request logs that your security team needs to audit frequently, you can store them in their own archive. This helps focus your searches only on the logs that are relevant to the investigation.
Datadog also allows you to cost-effectively manage your archives by letting you choose an optimal S3 storage class based on your requirements. For example, for critical archives that you rarely access, you can set up an S3 Lifecycle rule to automatically transition them to the S3 Glacier Instant Retrieval class archive, which could potentially cut your storage costs by up to 68 percent compared to using the older S3 Standard-Infrequent Access class. You can see our documentation for the full list of S3 storage classes that Log Rehydration supports.
Leave no log unturned
Logs are essential to getting insight into the overall health of your infrastructure, but common long-term log management solutions can make viewing older logs a challenge and using them for troubleshooting and analysis even harder. Datadog’s Log Rehydration™ allows you to resurface old logs for quick and efficient ad hoc investigations and analysis, without any additional tooling. Having easy access to all of your logs will help give you greater visibility into your applications and any of Datadog’s more than 800 integrations that you may be monitoring.
If you aren’t already using Datadog, get started with a 14-day free trial .
