Alerting is a critical component of your application and infrastructure monitoring plan. With the proper alerts in place you can anticipate issues before they fully develop or respond quickly to an isolated incident before it takes an application down.
Datadog’s alerts notify you when metrics cross thresholds you set; they have flexible conditions, appear in the event stream, and can be delivered to specific team members via email, HipChat, Slack or the person who is on call in PagerDuty or OpsGenie so you can take prompt action to solve an issue.
We are happy to introduce two new features to give you better control over your alert notifications: re-notifications and muted notifications for maintenance windows.
Get re-notified until an issue is resolved
For the most critical alerts you may want to be notified on a regular basis until the issue is fixed, as opposed to just once when the alert is triggered. You can think of it as a reminder option (or maybe more aptly as a snooze button).
You can now specify a re-notify time interval in the Alert Editor, so that the alert will notify periodically as long as the issue is not resolved.
For example, the alert below is set to re-notify every 40 minutes so when the user-defined threshold is crossed, an alert will trigger every 40 minutes until the issue is resolved.
Notifications will also show up as events in your event stream at the specified period until the issue is resolved.
Finally, you can specify an alert recipient using Datadog’s @ notification functionality which will send notifications emails or PagerDuty and OpsGenie alerts to the appropriate person or team.
Mute alert notifications during maintenance windows to reduce alerting noise
Let’s say that you are performing maintenance and you need to stop parts of your application, which would cause some alerts to (rightly) consider that something is wrong. You don’t want to delete those alerts or get notified during the maintenance window.
Datadog now enables you to mute any alert for preset durations to cover your maintenance window. Once the window has elapsed, muted alerts resume their usual notification business. To try it, head over to the Manage Alerts page and click on the “mute” button to pick a length of time you’d like to have the alert muted.
If you’re already using Datadog and have any questions about the alerting system, email us at support@datadoghq.com. If you’re not using Datadog and you’re interested in gaining more control over your alert notifications, sign up for a free 14 day Datadog trial and check out these new alerting features for yourself.
