AppFolio Selects Datadog LLM Observability to Monitor and Optimize Its AI Applications | Datadog
Case study

AppFolio selects Datadog LLM Observability to monitor and optimize its AI applications

Real Estate

1,500 
Employees

Goleta, 
California

AWS

About Appfolio

Appfolio is a real estate management software provider that supports customers in the property management industry. With 20,000+ customers and 8+ million units managed, AppFolio helps users maximize productivity, centralize data, and scale their business efficiently.

“Datadog LLM Observability helped us ensure high model performance and quality, and allowed us to expand functionality quickly and safely.”

case-studies/teddy_ho
Teddy Ho
Principal Product Manager
AppFolio
case-studies/teddy_ho

“Datadog LLM Observability helped us ensure high model performance and quality, and allowed us to expand functionality quickly and safely.”

Teddy Ho
Principal Product Manager
AppFolio
Why Datadog?
  • Enables company to monitor the performance and quality of its LLM-powered applications
  • Unified dashboards provide insights into LLM applications’ latency and token usage over time
  • Real-time alerts on evaluations help detect changes in application behavior to ensure optimal performance
Challenge

AppFolio wanted to better understand the non-determinism of LLMs and incorporate responsible AI practices while deploying and scaling LLM-powered applications. It also needed a solution that could integrate easily with Amazon Bedrock with minimal code annotations and identify potential issues and drive improvements during product development.

Key Results
80–90%

Reduction in latency

5 hours

Reduction in time spent per week on communications between property managers and residents

300%

Increase in product adoption

Identifying issues while building an AI-powered solution

AppFolio created a comprehensive platform to simplify and streamline property management and support clients at every stage of the resident lifecycle. The company’s powerful platform and centralized data framework enable its customers to manage all aspects of their operations, including tracking maintenance, recording payments, and linking expenses to financial documents.

While developing the platform, AppFolio discovered that its customers often dedicated up to 50 percent of their time communicating with residents of the properties they manage. Recognizing their desire to reduce time spent on correspondence so they could focus on other responsibilities and priorities, AppFolio’s leadership committed to building a generative artificial intelligence (AI)-powered application to facilitate and automate this communication.

appfolio_content_img_1.png

Building a new AI-powered app with confidence

AppFolio used Amazon Bedrock to build, test, and evaluate the LLM foundation models, customize them with data, and incorporate them into its messaging application. To support this project, AppFolio required a monitoring solution that would seamlessly integrate with Amazon Bedrock as well as identify potential issues and drive improvements as they developed the application. Additionally, the solution needed to uphold high quality standards as the company broadened its services.

AppFolio chose Datadog LLM Observability to monitor the performance and quality of its new application, which they named Realm-X Messages, an LLM-powered inbox that streamlines resident communications with smart organization and suggested actions. Datadog’s LLM Observability helps AppFolio track usage, performance, and error rates. Meanwhile, Datadog’s integrated dashboards offer insights into response times and token consumption over time, while real-time alerts detect anomalies, such as error spikes, to ensure optimal performance.

LLM Observability’s out-of-the-box evaluations of toxicity and failure-to-answer help the AppFolio team assess the quality of their LLM application’s responses and add necessary safeguards. In addition, the cluster map enables AppFolio to identify the various topics residents ask and how Realm-X performs on those topics using Datadog’s out-of-the-box evaluations. This further informs AppFolio’s R&D efforts and helps identify which features to prioritize. Given that hundreds of thousands of messages flow through AppFolio’s systems every day, having a solution to track, monitor, and analyze each interaction was critical.

With this comprehensive AI observability in place, the company was able to rapidly enhance its services and release new capabilities to an expanded customer base with confidence.

appfolio_content_img_2.png

Understanding, debugging, and evaluating usage and performance of LLM applications

After the initial setup, AppFolio was able to complete QA and move Realm-X Messages into production in less than a week.

“With LLM Observability, our team can understand, debug, and evaluate the usage and performance of our GenAI applications.”
Kyle TriplettVice President of Product, Appfolio

“We can monitor response quality to prevent negative interactions and ensure we’re providing our users with a positive experience.”

In the early stages of Realm-X Messages' release, AppFolio observed a strong correlation between reduced latency and adoption—as the service became faster, adoption increased. Using this as a target for improvement, AppFolio used LLM Observability to monitor the overall latency and identify the slowest traces. More importantly, AppFolio used LLM Observability to identify steps in its LLM chain—such as function calls, document retrieval, and calls to the LLM—that resulted in bottlenecks due to slow APIs and inefficient prompts. 

AppFolio used this insight to optimize Realm-X Messages’ architecture and application logic with updated prompts. This led to an 80 to 90 percent reduction in latency, which corresponded with a nearly 300 percent increase in the adoption of Realm-X Messages. “Datadog’s solution gave us confidence as we rolled out Realm-X Messages to more customers,” says Teddy Ho, Principal Product Manager for AppFolio. “It helped us ensure high performance and quality, and allowed us to expand functionality quickly and safely.”

Feedback from users has been positive. Realm-X Messages saves property managers an average of five hours per week on tasks related to communication, enabling them to spend less time on repetitive tasks and focus more on business growth. “Our priority is to constantly increase productivity gains for our customers,“ says Ho. “We want to make their jobs easier, and with Datadog’s help, we can now leverage generative AI to make that possible.”

Resources

products/llm-observability/llm-observability-product-hero-240612-desktop

official docs

LLM Observability
products/llm-observability/llm-observability-product-hero-240612-desktop

product

LLM Observability
/blog/datadog-llm-observability/llm-obs-hero

BLOG

Monitor, troubleshoot, improve, and secure your LLM applications with Datadog LLM Observability
/blog/llm-observability-chain-tracing/llm-obs-hero

BLOG

Get granular LLM observability by instrumenting your LLM chains