Identifying issues while building an AI-powered solution

AppFolio created a comprehensive platform to simplify and streamline property management and support clients at every stage of the resident lifecycle. The company’s powerful platform and centralized data framework enable its customers to manage all aspects of their operations, including tracking maintenance, recording payments, and linking expenses to financial documents.

While developing the platform, AppFolio discovered that its customers often dedicated up to 50 percent of their time communicating with residents of the properties they manage. Recognizing their desire to reduce time spent on correspondence so they could focus on other responsibilities and priorities, AppFolio’s leadership committed to building a generative artificial intelligence (AI)-powered application to facilitate and automate this communication.

Building a new AI-powered app with confidence

AppFolio used Amazon Bedrock to build, test, and evaluate the LLM foundation models, customize them with data, and incorporate them into its messaging application. To support this project, AppFolio required a monitoring solution that would seamlessly integrate with Amazon Bedrock as well as identify potential issues and drive improvements as they developed the application. Additionally, the solution needed to uphold high quality standards as the company broadened its services.

AppFolio chose Datadog LLM Observability to monitor the performance and quality of its new application, which they named Realm-X Messages, an LLM-powered inbox that streamlines resident communications with smart organization and suggested actions. Datadog’s LLM Observability helps AppFolio track usage, performance, and error rates. Meanwhile, Datadog’s integrated dashboards offer insights into response times and token consumption over time, while real-time alerts detect anomalies, such as error spikes, to ensure optimal performance.

LLM Observability’s out-of-the-box evaluations of toxicity and failure-to-answer help the AppFolio team assess the quality of their LLM application’s responses and add necessary safeguards. In addition, the cluster map enables AppFolio to identify the various topics residents ask and how Realm-X performs on those topics using Datadog’s out-of-the-box evaluations. This further informs AppFolio’s R&D efforts and helps identify which features to prioritize. Given that hundreds of thousands of messages flow through AppFolio’s systems every day, having a solution to track, monitor, and analyze each interaction was critical.

With this comprehensive AI observability in place, the company was able to rapidly enhance its services and release new capabilities to an expanded customer base with confidence.

Understanding, debugging, and evaluating usage and performance of LLM applications

After the initial setup, AppFolio was able to complete QA and move Realm-X Messages into production in less than a week.

“With LLM Observability, our team can understand, debug, and evaluate the usage and performance of our GenAI applications.”
Kyle TriplettVice President of Product, Appfolio

“We can monitor response quality to prevent negative interactions and ensure we’re providing our users with a positive experience.”

In the early stages of Realm-X Messages' release, AppFolio observed a strong correlation between reduced latency and adoption—as the service became faster, adoption increased. Using this as a target for improvement, AppFolio used LLM Observability to monitor the overall latency and identify the slowest traces. More importantly, AppFolio used LLM Observability to identify steps in its LLM chain—such as function calls, document retrieval, and calls to the LLM—that resulted in bottlenecks due to slow APIs and inefficient prompts. 

AppFolio used this insight to optimize Realm-X Messages’ architecture and application logic with updated prompts. This led to an 80 to 90 percent reduction in latency, which corresponded with a nearly 300 percent increase in the adoption of Realm-X Messages. “Datadog’s solution gave us confidence as we rolled out Realm-X Messages to more customers,” says Teddy Ho, Principal Product Manager for AppFolio. “It helped us ensure high performance and quality, and allowed us to expand functionality quickly and safely.”

Feedback from users has been positive. Realm-X Messages saves property managers an average of five hours per week on tasks related to communication, enabling them to spend less time on repetitive tasks and focus more on business growth. “Our priority is to constantly increase productivity gains for our customers,“ says Ho. “We want to make their jobs easier, and with Datadog’s help, we can now leverage generative AI to make that possible.”