Azure OpenAI is a service that helps you develop generative AI applications and custom copilots through OpenAI’s library of resources. With its easy-to-use REST APIs, you can leverage the service to access OpenAI’s powerful models, such as ChatGPT, for your applications while taking advantage of the reliability and security of the Azure platform. Datadog already offers an out-of-the-box integration for OpenAI so you can monitor key performance trends, such as API usage patterns, token consumption, and more. We’re proud to provide the same comprehensive visibility for Azure OpenAI, enabling you to monitor costs and performance issues that are specific to your Azure instances.
As Azure has continued to expand their OpenAI offerings, we’ve enhanced our existing Azure OpenAI integration in turn. With the latest updates to our integration, you can access additional out-of-the-box metrics alongside updated monitors and dashboards for even deeper visibility into model usage and performance. With Datadog’s Azure OpenAI integration, you can identify and troubleshoot performance degradations even faster, as well as track and fine-tune your usage costs on a more granular level.
Complete visibility into Azure OpenAI performance
Powered by Datadog’s existing Azure integration, our Azure OpenAI integration requires no additional setup. Once enabled, Datadog will start collecting metrics from your Azure OpenAI instances. You can use the built-in integration dashboard for an overview of performance across all your instances as well as usage trends for the Azure OpenAI service.
On this dashboard, you can view data such as the total number of executed API calls per Azure resource, the latency for these calls, and how many of them were blocked or rate limited. This enables you to quickly spot issues that could impact end-user experience. If calls for your application are regularly being denied due to rate limits, this may indicate that you need to increase the default limits or, alternatively, look into potential optimization strategies for your application.
For even deeper troubleshooting insights, you can also configure Datadog to collect Azure OpenAI logs, which provide you with the necessary context to identify the source of unusual changes in performance. For example, you can review your logs in Datadog Log Management to quickly determine the root cause of a sudden spike in errors for a particular Azure OpenAI instance.
Track usage and costs for your Azure OpenAI instances
Datadog’s Azure integration automatically collects performance data that is unique to Azure OpenAI instances. This ensures that you have the information you need to monitor usage and potential costs for all your AI applications, regardless of how they are deployed. For example, you can get a better picture of the costs for running your customized Azure OpenAI models by tracking the number of processed inference tokens and fine-tuned training hours per model deployment via the integration dashboard.
You can also use our out-of-the-box recommended monitors to automatically receive notifications on problematic usage cost activity. For example, you may want to be alerted when your Azure OpenAI token usage is abnormally high. This helps you quickly investigate and remediate the issue—which could be the result of an unpredictable spike in user traffic or new code added to your application—before you exceed the capacity of your existing resources. You can also use these monitors to detect user activity that may indicate security threats, such as API requests flagged by the Azure OpenAI content filter.
Start monitoring your Azure OpenAI applications today
Our out-of-the-box Azure integration gives you instant, comprehensive visibility into all of Azure’s supported language models and your Azure OpenAI instances. Check out our documentation to learn more about enabling the integration for your environment. If you don’t already have a Datadog account, you can sign up for a 14-day free trial today.
