# Evaluation and Continual Improvement for an Enterprise AI Assistant

arXiv: https://arxiv.org/abs/2407.12003 · HTML: https://mlsanity.com/paper/2407.12003/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2024.

## Built on by (most cited first, 7 of 7)

- [HierTOD: A Task-Oriented Dialogue System Driven by Hierarchical Goals](https://mlsanity.com/paper/2411.07152/) (2024)
- [Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations](https://mlsanity.com/paper/2601.22318/) (2026)
- [ELOQ: Resources for Enhancing LLM Detection of Out-of-Scope Questions](https://mlsanity.com/paper/2410.14567/) (2024)
- [Enhancing Discoverability in Enterprise Conversational Systems with Proactive Question Suggestions](https://mlsanity.com/paper/2412.10933/) (2024)
- [Evaluation and Incident Prevention in an Enterprise AI Assistant](https://mlsanity.com/paper/2504.13924/) (2025)
- [Adobe Summit Concierge Evaluation with Human in the Loop](https://mlsanity.com/paper/2511.03186/) (2025)
- [Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents](https://mlsanity.com/paper/2609.33955/) (2026)

## Abstract (arXiv)

The development of conversational AI assistants is an iterative process with multiple components. As such, the evaluation and continual improvement of these assistants is a complex and multifaceted problem. This paper introduces the challenges in evaluating and improving a generative AI assistant for enterprises, which is under active development, and how we address these challenges. We also share preliminary results and discuss lessons learned.