As organizations begin developing generative artificial intelligence (GenAI) applications, observability challenges could hinder their progress. Few robust monitoring tools for GenAI applications are available, which makes identifying and resolving issues in these applications time-consuming and error-prone. GenAI applications also often require multiple tools to observe the different aspects of the AI/Machine Learning (ML) tech stack, including infrastructure monitoring, ML job performance, and LLM app evaluation—and this fragmentation can lead to blind spots and inefficiencies. This inadequate observability ultimately leads to performance bottlenecks and a degraded user experience in GenAI applications.
In this blog, we’ll explore how Crest Data’s AI-focused integrations in the Datadog Marketplace can address these pain points by unifying different aspects of GenAI observability within the Datadog platform. Specifically, we’ll look at three examples of these integrations: Kong AI Gateway for AI infrastructure monitoring, Together.ai for ML job performance monitoring, and TruLens Eval for LLM app evaluation.
Crest Data’s AI-focused integrations in the Datadog Marketplace
Crest Data’s integrations with Datadog close the GenAI monitoring gap in areas such as infrastructure monitoring, ML job performance, and LLM app evaluation.
Infrastructure monitoring: Kong AI Gateway
To support GenAI applications, teams need logs and metrics in broad infrastructure areas such as API traffic, anomaly detection in API usage, and overall AI infrastructure performance. MLOps engineers need this data to help ensure that infrastructure is running smoothly, to identify any potential performance issues, and to enhance security.
The Kong AI Gateway integration with Datadog helps users track real-time GenAI application traffic metrics and identify unusual patterns that could indicate performance issues or security threats. Engineers can also use this integration to monitor API performance metrics that measure response times, error rates, and throughput. In the event of an incident, the integration allows these engineers to quickly pinpoint root causes by correlating API traffic anomalies with infrastructure metrics. These capabilities help teams resolve issues faster and minimize downtime.
ML job performance: Together.ai
With the help of this integration from Crest Data, Together.ai can send data such as ML fine-tuning job performance metrics, model usage data, job event details, and training configuration data to Datadog. Once this data is collected, the integration allows engineers to monitor and optimize ML job performance and track proprietary data files used for fine-tuning. This helps engineers glean insights into model status, usage, and training performance. If issues are found as a result of the fine-tuning jobs, the integration allows you to quickly identify the root cause and implement fixes.
LLM app evaluation: TruLens Eval
The TruLens integration by Crest Data provides performance metrics and detailed evaluation reports to help you understand LLM app performance, identify potential issues, and optimize model accuracy and efficiency. You can also evaluate your LLM apps with feedback functions such as groundedness, answer relevance, fairness, and bias to increase performance and minimize risk.
Making this data available in the Datadog platform helps provide additional in-depth monitoring of GenAI model evaluation and performance. For example, users can leverage Datadog’s dashboarding capabilities to visualize model metrics, set up alerts for performance anomalies, and analyze detailed evaluation reports. By configuring alerts that indicate whenever a deviation occurs from expected performance values, you can quickly identify data drift or model degradation, allowing you to better maintain model efficacy.
Monitor more of your Gen AI stack with Datadog and Crest Data
Crest Data’s AI-focused integrations enable teams to improve their AI infrastructure monitoring, ML job performance monitoring, and LLM app evaluation. And because the integrations combine these capabilities within the Datadog platform, organizations can overcome the challenges of fragmented AI monitoring to achieve greater performance in their GenAI applications.
Crest Data’s integrations are available in the Datadog Marketplace. The ability to develop and promote third-party tools in the Datadog Marketplace that extend the capabilities of Datadog is one of the benefits of membership in the Datadog Partner Network. You can learn more about the Datadog Marketplace in our blog post about this topic or in our documentation. You can also contact us at marketplace@datadog.com if you’re interested in developing an integration or application.
