Data build tool (dbt) is an open source service that cleans, aggregates, and models raw data into organized, analytics-ready formats within a data warehouse. dbt Cloud, a fully managed platform by dbt Labs, extends dbt’s capabilities with advanced features such as scheduling, testing, and monitoring, accessible directly from your browser.
While dbt Cloud relieves you of having to manually handle data transformation tasks, you still need visibility into your automated workflows to ensure that jobs are performing as expected, and that you can quickly identify and resolve failed jobs to prevent downstream issues. Maintaining that visibility becomes particularly challenging at scale as the number of jobs, models, and users grows.
In this post, we’ll look at how Datadog’s dbt Cloud integration provides you with all the visibility you need to:
- Monitor the performance and health of your dbt runs
- Visualize model execution and compilation times, as well as test run results
Monitor the performance and health of your dbt job runs
Jobs in dbt are predefined, scheduled tasks that automate the execution of the dbt models in your project. These jobs are composed of multiple dbt commands that are chained together and configured to execute in a specified sequence. For example, let’s say you scheduled a data pipeline job to run every morning. This job may first execute a dbt run command that builds and transforms data models and then a dbt test command that validates their quality. This ensures your data pipeline is up to date and reliable, catching issues like null values or duplicates before the workday begins.
To ensure smooth data pipeline operations, jobs must run fast and reliably. Slow runs can create bottlenecks, cause problems with downstream systems, and delay critical data analysis. Datadog’s dbt Cloud integration includes an out-of-the-box dashboard that provides a complete overview of all the job runs in your dbt project, enabling you to quickly check on their status and performance.
This dashboard includes a top list of the longest-running dbt jobs, helping you quickly pinpoint which job might be behind a bottleneck and take steps toward mitigation. For instance, if you notice a specific job takes significantly longer than others, you can review its run history. This could reveal issues like inefficient queries, large data volumes, or unnecessary full refreshes, allowing you to address the root cause and improve performance.
In addition to runtime insights, our dbt Cloud integration also collects and highlights job statuses so you can immediately see which jobs were successful, which have been queued, and which have failed or are experiencing errors.
Visualize model execution and compilation times and test run results
In dbt, models are SQL or Python files that contain the logic for transforming raw data into usable formats for reporting and analysis. They define how data should be queried and transformed and serve as core components of dbt projects. Models in dbt need to compile and execute quickly to ensure efficient development, as fast compilation and execution times enhance developer productivity by enabling rapid feedback loops, real-time debugging, and easier error detection. Fast dbt models also help minimize delays in CI/CD workflows and data pipelines, reducing costs and improving time-sensitive processes.
Our dbt Cloud integration collects model execution and compilation time metrics so you can ensure optimal performance. Tracking these metrics and setting alerts on slow compilation and execution times will help you identify models that are taking longer than expected. You can then take steps to fine-tune queries, optimize dependencies, and maintain a scalable, reliable data pipeline.
To ensure your models are behaving as expected, dbt allows you to define and run tests that validate their outputs against specific rules. The dbt Cloud dashboard visualizes test results and reports on test success rate, allowing you to quickly identify trends, pinpoint failing tests, and monitor the overall health of your models.
Start monitoring dbt Cloud today
With our dbt Cloud integration, DevOps teams can gain insights into the health and performance of their dbt runs, models, and tests as they bring their data warehouse transformation processes to scale. And with Datadog’s 850+ other integrations, teams can monitor dbt Cloud alongside telemetry from across their stack––all on a single, unified platform.
Use this link to log into your Datadog instance and get started with our dbt Cloud integration today. Not a Datadog customer? Get started with a 14-day free trial.
