mlsanityData to 14 Oct 2026

Exploring the Impact of Human Evaluator Group on Chat-Oriented Dialogue Evaluation

arXiv 2309.07998 ↗First posted 14 Sep 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Human evaluation has been widely accepted as the standard for evaluating chat-oriented dialogue systems. However, there is a significant variation in previous work regarding who gets recruited as evaluators. Evaluator groups such as domain experts, university students, and professional annotators have been used to assess and compare dialogue systems, although it is unclear to what extent the choice of an evaluator group can affect results. This paper analyzes the evaluator group impact on dialogue system evaluation by testing 4 state-of-the-art dialogue systems using 4 distinct evaluator groups. Our analysis reveals a robustness towards evaluator groups for Likert evaluations that is not seen for Pairwise, with only minor differences observed when changing evaluator groups. Furthermore, two notable limitations to this robustness are observed, which reveal discrepancies between evaluators with different levels of chatbot expertise and indicate that evaluator objectivity is beneficial for certain dialogue metrics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

18 of 27 references.

References · 18

  1. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  2. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  3. 2021Towards Emotional Support Dialog Systemscited by 299
  4. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  5. 2021BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Datacited by 75
  6. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  7. 2018Conversational AI: The Science Behind the Alexa Prizecited by 63
  8. 2018Importance of Search and Evaluation Strategies in Neural Dialogue Modelingcited by 44
  9. 2021Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causescited by 43
  10. 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
  11. 2022Achieving Reliable Human Assessment of Open-Domain Dialogue Systemscited by 31
  12. 2022A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generationcited by 25
6 more
  1. 2020Emora: An Inquisitive Social Chatbot Who Cares For Youcited by 16
  2. 2022HeterMPC: A Heterogeneous Graph Neural Network for Response Generation in Multi-Party Conversationscited by 16
  3. 2021Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Trainingcited by 13
  4. 2020An Evaluation Protocol for Generative Conversational Systemscited by 9
  5. 2022Commonsense and Named Entity Aware Knowledge Grounded Dialogue Generationcited by 8
  6. 2022Probing the Robustness of Trained Metrics for Conversational Dialogue Systemscited by 6

Built on by

Not cited yet.