# Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols

arXiv: https://arxiv.org/abs/2006.06110 · HTML: https://mlsanity.com/paper/2006.06110/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2020.

## Built on by (most cited first, 30 of 35)

- [ProsocialDialog: A Prosocial Backbone for Conversational Agents](https://mlsanity.com/paper/2205.12688/) (2022)
- [Keep Me Updated! Memory Management in Long-term Conversations](https://mlsanity.com/paper/2210.08750/) (2022)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [DynaEval: Unifying Turn and Dialogue Level Evaluation](https://mlsanity.com/paper/2106.01112/) (2021)
- [Rewarding Chatbots for Real-World Engagement with Millions of Users](https://mlsanity.com/paper/2303.06135/) (2023)
- [Achieving Reliable Human Assessment of Open-Domain Dialogue Systems](https://mlsanity.com/paper/2203.05899/) (2022)
- [Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models](https://mlsanity.com/paper/2205.00176/) (2022)
- [Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach](https://mlsanity.com/paper/2102.10242/) (2021)
- [Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges](https://mlsanity.com/paper/2203.10012/) (2022)
- [LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing](https://mlsanity.com/paper/2105.01992/) (2021)
- [ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems](https://mlsanity.com/paper/2305.07797/) (2023)
- [Code-switched inspired losses for generic spoken dialog representations](https://mlsanity.com/paper/2108.12465/) (2021)
- [Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems](https://mlsanity.com/paper/2212.09180/) (2022)
- [Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations](https://mlsanity.com/paper/2312.13871/) (2023)
- [Mixed-Session Conversation with Egocentric Memory](https://mlsanity.com/paper/2410.02503/) (2024)
- [Human-like informative conversations: Better acknowledgements using conditional mutual information](https://mlsanity.com/paper/2104.07831/) (2021)
- [Towards Standard Criteria for human evaluation of Chatbots: A Survey](https://mlsanity.com/paper/2105.11197/) (2021)
- [TableTalk: Scaffolding Spreadsheet Development with a Language Agent](https://mlsanity.com/paper/2502.09787/) (2025)
- [Task-oriented Dialogue Systems: performance vs. quality-optima, a review](https://mlsanity.com/paper/2112.11176/) (2021)
- [What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts](https://mlsanity.com/paper/2111.00572/) (2021)
- [SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation](https://mlsanity.com/paper/2208.08094/) (2022)
- [Actionable Conversational Quality Indicators for Improving Task-Oriented Dialog Systems](https://mlsanity.com/paper/2109.11064/) (2021)
- [Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems](https://mlsanity.com/paper/2205.12554/) (2022)
- [A Comprehensive Survey of Natural Language Generation Advances from the Perspective of Digital Deception](https://mlsanity.com/paper/2208.05757/) (2022)
- [A-CAP: Anticipation Captioning with Commonsense Knowledge](https://mlsanity.com/paper/2304.06602/) (2023)
- [It Couldn't Help But Overhear: On the Limits of Modelling Meta-Communicative Grounding Acts with Supervised Learning](https://mlsanity.com/paper/2405.01139/) (2024)
- [Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey](https://mlsanity.com/paper/2411.09166/) (2024)
- [ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisors](https://mlsanity.com/paper/2608.01204/) (2026)
- [Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight](https://mlsanity.com/paper/2608.24314/) (2026)
- [Relevance in Dialogue: Is Less More? An Empirical Comparison of Existing Metrics, and a Novel Simple Metric](https://mlsanity.com/paper/2206.01823/) (2022)

## Abstract (arXiv)

As conversational AI-based dialogue management has increasingly become a trending topic, the need for a standardized and reliable evaluation procedure grows even more pressing. The current state of affairs suggests various evaluation protocols to assess chat-oriented dialogue management systems, rendering it difficult to conduct fair comparative studies across different approaches and gain an insightful understanding of their values. To foster this research, a more robust evaluation protocol must be set in place. This paper presents a comprehensive synthesis of both automated and human evaluation methods on dialogue systems, identifying their shortcomings while accumulating evidence towards the most effective evaluation dimensions. A total of 20 papers from the last two years are surveyed to analyze three types of evaluation protocols: automated, static, and interactive. Finally, the evaluation dimensions used in these papers are compared against our expert evaluation on the system-user dialogue data collected from the Alexa Prize 2020.