# Achieving Reliable Human Assessment of Open-Domain Dialogue Systems

arXiv: https://arxiv.org/abs/2203.05899 · HTML: https://mlsanity.com/paper/2203.05899/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 46 references that are in the map.
- First posted on arXiv 11 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons](https://mlsanity.com/paper/1909.03087/) (2019)
- [RankME: Reliable Human Ratings for Natural Language Generation](https://mlsanity.com/paper/1803.05928/) (2018)
- [Conversational AI: The Science Behind the Alexa Prize](https://mlsanity.com/paper/1801.03604/) (2018)
- [Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols](https://mlsanity.com/paper/2006.06110/) (2020)
- [Towards Best Experiment Design for Evaluating Dialogue System Output](https://mlsanity.com/paper/1909.10122/) (2019)
- [Studying the Effects of Cognitive Biases in Evaluation of Conversational Agents](https://mlsanity.com/paper/2002.07927/) (2020)

## Built on by (most cited first, 30 of 31)

- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [Keep Me Updated! Memory Management in Long-term Conversations](https://mlsanity.com/paper/2210.08750/) (2022)
- [A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators](https://mlsanity.com/paper/2312.15407/) (2023)
- [Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2402.13374/) (2024)
- [RevisEval: Improving LLM-as-a-Judge via Response-Adapted References](https://mlsanity.com/paper/2410.05193/) (2024)
- [A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models](https://mlsanity.com/paper/2305.01181/) (2023)
- [PRODIGy: a PROfile-based DIalogue Generation dataset](https://mlsanity.com/paper/2311.05195/) (2023)
- [CoAScore: Chain-of-Aspects Prompting for NLG Evaluation](https://mlsanity.com/paper/2312.10355/) (2023)
- [xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark](https://mlsanity.com/paper/2310.08958/) (2023)
- [SimOAP: Improve Coherence and Consistency in Persona-based Dialogue Generation via Over-sampling and Post-evaluation](https://mlsanity.com/paper/2305.11130/) (2023)
- [QAScore -- An Unsupervised Unreferenced Metric for the Question Generation Evaluation](https://mlsanity.com/paper/2210.04320/) (2022)
- [Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems](https://mlsanity.com/paper/2212.09180/) (2022)
- [RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue](https://mlsanity.com/paper/2309.08156/) (2023)
- [Role-Play Zero-Shot Prompting with Large Language Models for Open-Domain Human-Machine Conversation](https://mlsanity.com/paper/2406.18460/) (2024)
- [State-of-the-art in Open-domain Conversational AI: A Survey](https://mlsanity.com/paper/2205.00965/) (2022)
- [Communication is All You Need: Persuasion Dataset Construction via Multi-LLM Communication](https://mlsanity.com/paper/2502.08896/) (2025)
- [Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs](https://mlsanity.com/paper/2306.03984/) (2023)
- [HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants](https://mlsanity.com/paper/2405.09186/) (2024)
- [ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmark](https://mlsanity.com/paper/2406.11228/) (2024)
- [Building Multimodal AI Chatbots](https://mlsanity.com/paper/2305.03512/) (2023)
- [Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversations](https://mlsanity.com/paper/2509.00841/) (2025)
- [A Multilingual Virtual Guide for Self-Attachment Technique](https://mlsanity.com/paper/2310.18366/) (2023)
- [Findings of the First Workshop on Simulating Conversational Intelligence in Chat](https://mlsanity.com/paper/2402.06420/) (2024)
- [OnRL-RAG: Real-Time Personalized Mental Health Dialogue System](https://mlsanity.com/paper/2504.02894/) (2025)
- [Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Error](https://mlsanity.com/paper/2512.16750/) (2025)
- [Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight](https://mlsanity.com/paper/2608.24314/) (2026)
- [Reranking Overgenerated Responses for End-to-End Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2211.03648/) (2022)
- [Exploring the Impact of Human Evaluator Group on Chat-Oriented Dialogue Evaluation](https://mlsanity.com/paper/2309.07998/) (2023)
- [Rethinking Response Evaluation from Interlocutor's Eye for Open-Domain Dialogue Systems](https://mlsanity.com/paper/2401.02256/) (2024)
- [UniRQR: A Unified Model for Retrieval Decision, Query, and Response Generation in Internet-Based Knowledge Dialogue Systems](https://mlsanity.com/paper/2401.06811/) (2024)

## Abstract (arXiv)

Evaluation of open-domain dialogue systems is highly challenging and development of better techniques is highlighted time and again as desperately needed. Despite substantial efforts to carry out reliable live evaluation of systems in recent competitions, annotations have been abandoned and reported as too unreliable to yield sensible results. This is a serious problem since automatic metrics are not known to provide a good indication of what may or may not be a high-quality conversation. Answering the distress call of competitions that have emphasized the urgent need for better evaluation techniques in dialogue, we present the successful development of human evaluation that is highly reliable while still remaining feasible and low cost. Self-replication experiments reveal almost perfectly repeatable results with a correlation of r=0.969. Furthermore, due to the lack of appropriate methods of statistical significance testing, the likelihood of potential improvements to systems occurring due to chance is rarely taken into account in dialogue evaluation, and the evaluation we propose facilitates application of standard tests. Since we have developed a highly reliable evaluation method, new insights into system performance can be revealed. We therefore include a comparison of state-of-the-art models (i) with and without personas, to measure the contribution of personas to conversation quality, as well as (ii) prescribed versus freely chosen topics. Interestingly with respect to personas, results indicate that personas do not positively contribute to conversation quality as expected.