Understanding, debugging, and evaluating usage and performance of LLM applications
After the initial setup, AppFolio was able to complete QA and move Realm-X Messages into production in less than a week.
“With LLM Observability, our team can understand, debug, and evaluate the usage and performance of our GenAI applications.”
Kyle TriplettVice President of Product, Appfolio
“We can monitor response quality to prevent negative interactions and ensure we’re providing our users with a positive experience.”
In the early stages of Realm-X Messages' release, AppFolio observed a strong correlation between reduced latency and adoption—as the service became faster, adoption increased. Using this as a target for improvement, AppFolio used LLM Observability to monitor the overall latency and identify the slowest traces. More importantly, AppFolio used LLM Observability to identify steps in its LLM chain—such as function calls, document retrieval, and calls to the LLM—that resulted in bottlenecks due to slow APIs and inefficient prompts.
AppFolio used this insight to optimize Realm-X Messages’ architecture and application logic with updated prompts. This led to an 80 to 90 percent reduction in latency, which corresponded with a nearly 300 percent increase in the adoption of Realm-X Messages. “Datadog’s solution gave us confidence as we rolled out Realm-X Messages to more customers,” says Teddy Ho, Principal Product Manager for AppFolio. “It helped us ensure high performance and quality, and allowed us to expand functionality quickly and safely.”
Feedback from users has been positive. Realm-X Messages saves property managers an average of five hours per week on tasks related to communication, enabling them to spend less time on repetitive tasks and focus more on business growth. “Our priority is to constantly increase productivity gains for our customers,“ says Ho. “We want to make their jobs easier, and with Datadog’s help, we can now leverage generative AI to make that possible.”