Datadog is a central hub of information—enabling you to see logs, traces, and metrics from across your stack and providing a centralized source of notifications about potential issues. However, when Datadog notifies you of an issue, you often need to log in to other applications to fully assess and resolve it, which slows down mitigation. Datadog’s Workflow Automation and App Builder enable you to act on insights directly within Datadog, saving time during critical moments when you need it the most. To make this possible, Workflow Automation and App Builder have more than 1000 built-in actions covering all major cloud providers and popular tools (such as GitHub, GitLab, JIRA, and Status Page) that seamlessly connect Datadog to the rest of your stack.
Workflows and apps that include custom JavaScript code are now even easier to build by any member of your team—including those not comfortable writing code or who lack the time to dig around in Datadog for the best way to parse tags or reference variables. Datadog’s new AI assistant for Workflow Automation and App Builder uses generative AI to automate much of the code-writing process. Drawing on its knowledge of the current workflow and the actions in the Datadog Actions Catalog, the AI assistant can parse telemetry data, suggest code to accomplish a step in a workflow, troubleshoot code, and more.
This post walks through the process of building a workflow that restarts a service experiencing high error rates, showing you how you can use the new AI assistant to add complex JavaScript functions without any coding knowledge.
Create a workflow that restarts a service experiencing high error rates
Let’s say you want to create a workflow that restarts a service experiencing high error rates. You already have a monitor set up for the service, as well as a GitHub Actions workflow that restarts it. When error rates reach a certain threshold, the workflow will be triggered. It will first seek approval for the restart by posting in a designated Slack channel. Once it receives approval, it will restart the service via GitHub Actions.
To start building this workflow, head to the Workflows page and select +New Workflow in the top right corner. Next, choose your trigger type. You want this workflow to be automatically triggered when an alert goes off, so you’ll select Monitor under Datadog Triggers.
Before the workflow takes action, you want to add a flow where it sends a message to a Slack channel asking for approval. To do that, add a “Make a decision” Slack approval step.
The resulting Slack message provides built-in Approve and Reject buttons so the user can easily take action.
Because this is a native Datadog trigger, you automatically have access to all of the monitor’s metadata, including the monitor message and tags. To view all of the metadata that’s available to your workflow, save the workflow and click Run in the top left corner to perform a quick test.
To see the metadata associated with the monitor, expand the monitor object under Source in the right-hand panel.
Use the AI assistant to add JavaScript code to an approval step
You don’t want to ask just anyone to approve the restart—you want to ensure that the approver is in a set list of users. To add this check, you need to add JavaScript code, and you’ll use the new AI assistant to help you write it.
Add a JavaScript action to the workflow and click “Write Code with AI” to open the AI assistant, which can write fully functional JavaScript code based on your written prompts.
Type the prompt, “let’s check to see if the slack username is in an approved list.” The AI assistant provides code that will perform this check and gives you the option to replace the script, insert it, or copy it to the clipboard. To make this code functional, you just need to swap out the list of names (user1, user2, user3) with the Slack usernames of the people who are allowed to review.
You can use the AI assistant to write code as simple as “return today’s date in the format yyyy-MM-dd_hh:mm." But it is particularly useful for writing more complex functions, because it is aware of the context of the current workflow and can understand the API outputs from all of the actions in the Datadog Actions Catalog. In the above example, it automatically knows how to reference the username of the Slack user who responded to the approval request.
You can also ask the AI assistant to explain the logic behind code or pinpoint errors by typing “What is this code doing?” or “What is wrong with this code?”
The AI assistant can also help you parse information from the monitor if you don’t want to look through the monitor object to find a correct variable path. For example, the prompt “Give me all the team tag values from the monitor” finds all the teams associated with the monitor, which can be helpful if you want to slack a particular team for an approval step.
Take action once approved
After confirming that the approver is in the allowlist, the next step is to take action to remedy the issue. Some of the remediation actions available in Workflow Automation include restarting a VM, rebooting an EC2 instance, toggling feature flags, restarting a Kubernetes deployment, and kicking off automations built in other platforms like Jenkins and GitHub Actions. You can also make custom HTTP requests to any public or private endpoint that Datadog does not natively support.
In this case, you will use a GitHub Action workflow that you’ve already built in GitHub. To add this to the workflow, you would simply select “Trigger GitHub Actions Workflow Run” as the next step.
Before you publish your workflow, you can complete some test runs to ensure that everything is configured correctly. After it is published, the workflow can be added to any monitor in the Trigger Workflow panel in the “Configure notifications & automations” section of the monitor’s settings.
Use Datadog App Builder to create a self-service workflow
To enable self-service so your team can trigger the service restart manually, you can use App Builder to build an app that triggers the same GitHub Actions workflow. App Builder offers many blueprints that you can use as starting points, including “Restart Service with GitHub Actions.” Apps can be added to any dashboard or launched from their own dedicated page in Datadog.
You can use the new AI assistant to transform data in App Builder apps, just like you can in workflows. Apps can be built to run actions directly or to trigger any Datadog workflow. For example, if you’ve already created a workflow for a complex approvals flow, you can configure your app to run that workflow instead of simply performing a single action.
Get started using Workflow Automation and App Builder today
Workflow Automation, App Builder, and our new AI assistant are generally available to all Datadog customers. Now you can start building workflows and apps to kick off automated remediation in response to Datadog alerts, with little to no prior coding knowledge. You can include human approvals or perform the action directly, based on your comfort level, and add additional steps to adhere to any internal processes your team currently follows.
To learn more, see our Workflow Automation documentation and blueprints, as well as our App Builder documentation and blueprints. If you aren’t yet a Datadog customer, sign up for a 14-day free trial today.