# On Evaluating and Comparing Open Domain Dialog Systems

arXiv: https://arxiv.org/abs/1801.03625 · HTML: https://mlsanity.com/paper/1801.03625/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2018.

## Built on by (most cited first, 16 of 16)

- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering](https://mlsanity.com/paper/2408.13545/) (2024)
- [Evaluating LLM Metrics Through Real-World Capabilities](https://mlsanity.com/paper/2505.08253/) (2025)
- [HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations](https://mlsanity.com/paper/2106.00162/) (2021)
- [Jurassic is (almost) All You Need: Few-Shot Meaning-to-Text Generation for Open-Domain Dialogue](https://mlsanity.com/paper/2110.08094/) (2021)
- [Modeling Performance in Open-Domain Dialogue with PARADISE](https://mlsanity.com/paper/2110.11164/) (2021)
- [State-of-the-art in Open-domain Conversational AI: A Survey](https://mlsanity.com/paper/2205.00965/) (2022)
- [Hear You in Silence: Designing for Active Listening in Human Interaction with Conversational Agents Using Context-Aware Pacing](https://mlsanity.com/paper/2602.06134/) (2026)
- [A Conversational Agent System for Dietary Supplements Use](https://mlsanity.com/paper/2104.01543/) (2021)
- [How GPT-3 responds to different publics on climate change and Black Lives Matter: A critical appraisal of equity in conversational AI](https://mlsanity.com/paper/2209.13627/) (2022)
- [Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversations](https://mlsanity.com/paper/2509.00841/) (2025)
- [LLM-Mediated Domain-Specific Voice Agents: The Case of TextileBot](https://mlsanity.com/paper/2406.10590/) (2024)
- [Conversational Agents for Older Adults' Health: A Systematic Literature Review](https://mlsanity.com/paper/2503.23153/) (2025)
- [Dialogue Object Search](https://mlsanity.com/paper/2107.10653/) (2021)

## Abstract (arXiv)

Conversational agents are exploding in popularity. However, much work remains in the area of non goal-oriented conversations, despite significant growth in research interest over recent years. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million dollar university competition where sixteen selected university teams built conversational agents to deliver the best social conversational experience. Alexa Prize provided the academic community with the unique opportunity to perform research with a live system used by millions of users. The subjectivity associated with evaluating conversations is key element underlying the challenge of building non-goal oriented dialogue systems. In this paper, we propose a comprehensive evaluation strategy with multiple metrics designed to reduce subjectivity by selecting metrics which correlate well with human judgement. The proposed metrics provide granular analysis of the conversational agents, which is not captured in human ratings. We show that these metrics can be used as a reasonable proxy for human judgment. We provide a mechanism to unify the metrics for selecting the top performing agents, which has also been applied throughout the Alexa Prize competition. To our knowledge, to date it is the largest setting for evaluating agents with millions of conversations and hundreds of thousands of ratings from users. We believe that this work is a step towards an automatic evaluation process for conversational AIs.