# Exploring the Impact of Human Evaluator Group on Chat-Oriented Dialogue Evaluation

arXiv: https://arxiv.org/abs/2309.07998 · HTML: https://mlsanity.com/paper/2309.07998/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 18 of 18)

- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons](https://mlsanity.com/paper/1909.03087/) (2019)
- [BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data](https://mlsanity.com/paper/2106.06169/) (2021)
- [I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling](https://mlsanity.com/paper/2012.13391/) (2020)
- [Conversational AI: The Science Behind the Alexa Prize](https://mlsanity.com/paper/1801.03604/) (2018)
- [Importance of Search and Evaluation Strategies in Neural Dialogue Modeling](https://mlsanity.com/paper/1811.00907/) (2018)
- [Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://mlsanity.com/paper/2109.08828/) (2021)
- [Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols](https://mlsanity.com/paper/2006.06110/) (2020)
- [Achieving Reliable Human Assessment of Open-Domain Dialogue Systems](https://mlsanity.com/paper/2203.05899/) (2022)
- [A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation](https://mlsanity.com/paper/2204.09867/) (2022)
- [Emora: An Inquisitive Social Chatbot Who Cares For You](https://mlsanity.com/paper/2009.04617/) (2020)
- [HeterMPC: A Heterogeneous Graph Neural Network for Response Generation in Multi-Party Conversations](https://mlsanity.com/paper/2203.08500/) (2022)
- [Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training](https://mlsanity.com/paper/2105.15171/) (2021)
- [An Evaluation Protocol for Generative Conversational Systems](https://mlsanity.com/paper/2010.12741/) (2020)
- [Commonsense and Named Entity Aware Knowledge Grounded Dialogue Generation](https://mlsanity.com/paper/2205.13928/) (2022)
- [Probing the Robustness of Trained Metrics for Conversational Dialogue Systems](https://mlsanity.com/paper/2202.13887/) (2022)

## Abstract (arXiv)

Human evaluation has been widely accepted as the standard for evaluating chat-oriented dialogue systems. However, there is a significant variation in previous work regarding who gets recruited as evaluators. Evaluator groups such as domain experts, university students, and professional annotators have been used to assess and compare dialogue systems, although it is unclear to what extent the choice of an evaluator group can affect results. This paper analyzes the evaluator group impact on dialogue system evaluation by testing 4 state-of-the-art dialogue systems using 4 distinct evaluator groups. Our analysis reveals a robustness towards evaluator groups for Likert evaluations that is not seen for Pairwise, with only minor differences observed when changing evaluator groups. Furthermore, two notable limitations to this robustness are observed, which reveal discrepancies between evaluators with different levels of chatbot expertise and indicate that evaluator objectivity is beneficial for certain dialogue metrics.