# On the Evaluation of Dialogue Systems with Next Utterance Classification

arXiv: https://arxiv.org/abs/1605.05414 · HTML: https://mlsanity.com/paper/1605.05414/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2016.

## Built on by (most cited first, 8 of 8)

- [Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions](https://mlsanity.com/paper/2106.03873/) (2021)
- [Automatic Text Evaluation through the Lens of Wasserstein Barycenters](https://mlsanity.com/paper/2108.12463/) (2021)
- [Code-switched inspired losses for generic spoken dialog representations](https://mlsanity.com/paper/2108.12465/) (2021)
- [Meta-evaluation of Conversational Search Evaluation Metrics](https://mlsanity.com/paper/2104.13453/) (2021)
- [A Survey of Personality, Persona, and Profile in Conversational Agents and Chatbots](https://mlsanity.com/paper/2401.00609/) (2023)
- [Semantic Frame Forecast](https://mlsanity.com/paper/2104.05604/) (2021)
- [On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems](https://mlsanity.com/paper/2104.06335/) (2021)
- [From Knowledge Augmentation to Multi-tasking: Towards Human-like Dialogue Systems](https://mlsanity.com/paper/2212.03279/) (2022)

## Abstract (arXiv)

An open challenge in constructing dialogue systems is developing methods for automatically learning dialogue strategies from large amounts of unlabelled data. Recent work has proposed Next-Utterance-Classification (NUC) as a surrogate task for building dialogue systems from text data. In this paper we investigate the performance of humans on this task to validate the relevance of NUC as a method of evaluation. Our results show three main findings: (1) humans are able to correctly classify responses at a rate much better than chance, thus confirming that the task is feasible, (2) human performance levels vary across task domains (we consider 3 datasets) and expertise levels (novice vs experts), thus showing that a range of performance is possible on this type of task, (3) automated dialogue systems built using state-of-the-art machine learning methods have similar performance to the human novices, but worse than the experts, thus confirming the utility of this class of tasks for driving further research in automated dialogue systems.