# A Methodology for Identifying Evaluation Items for Practical Dialogue Systems Based on Business-Dialogue System Alignment Models

arXiv: https://arxiv.org/abs/2602.15835 · HTML: https://mlsanity.com/paper/2602.15835/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 60 references that are in the map.
- First posted on arXiv 10 Jan 2026.

## Builds on (most cited first, 12 of 12)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems](https://mlsanity.com/paper/2204.03021/) (2022)
- [Ethical Challenges in Data-Driven Dialogue Systems](https://mlsanity.com/paper/1711.09050/) (2017)
- [On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark](https://mlsanity.com/paper/2110.08466/) (2021)
- [Are LLMs All You Need for Task-Oriented Dialogue?](https://mlsanity.com/paper/2304.06556/) (2023)
- [What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents](https://mlsanity.com/paper/1901.06525/) (2019)
- [Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances](https://mlsanity.com/paper/2204.10825/) (2022)

## Abstract (arXiv)

This paper proposes a methodology for identifying evaluation items for practical dialogue systems. Traditionally, user satisfaction and user experiences have been the primary metrics for evaluating dialogue systems. However, there are various other evaluation items to consider when developing and operating practical dialogue systems, and such evaluation items are expected to lead to new research topics. So far, there has been no methodology for identifying these evaluation items. We propose identifying evaluation items based on business-dialogue system alignment models, which are applications of business-IT alignment models used in the development and operation of practical IT systems. We also present a generic model that facilitates the construction of a business-dialogue system alignment model for each dialogue system.