mlsanityData to 7 Oct 2026

Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents

arXiv 2201.04723 ↗First posted 12 Jan 2022cs.CL cs.AIcited by 62

Not on the map. Why?

Abstract (arXiv)

At the heart of improving conversational AI is the open problem of how to evaluate conversations. Issues with automatic metrics are well known (Liu et al., 2016, arXiv:1603.08023), with human evaluations still considered the gold standard. Unfortunately, how to perform human evaluations is also an open problem: differing data collection methods have varying levels of human agreement and statistical sensitivity, resulting in differing amounts of human annotation hours and labor costs. In this work we compare five different crowdworker-based human evaluation methods and find that different methods are best depending on the types of models compared, with no clear winner across the board. While this highlights the open problems in the area, our analysis leads to advice of when to use which one, and possible future directions.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 39 of its 60 references that are in the map. First posted on arXiv 12 Jan 2022.

Builds on

39 of 60 references.

References · 39

  1. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  2. 2020Recipes for building an open-domain chatbotcited by 593
  3. 2019ELI5: Long Form Question Answeringcited by 567
  4. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  5. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  6. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  7. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  8. 2020The Pushshift Reddit Datasetcited by 289
  9. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  10. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  11. 2016A Persona-Based Neural Conversation Modelcited by 221
  12. 2021Internet-Augmented Dialogue Generationcited by 217
27 more
  1. 2015A Neural Conversational Modelcited by 195
  2. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  3. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  4. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  5. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  6. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  7. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  8. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  9. 2016Learning End-to-End Goal-Oriented Dialogcited by 104
  10. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  11. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  12. 2017Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systemscited by 84
  13. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  14. 2018Conversational AI: The Science Behind the Alexa Prizecited by 63
  15. 2019Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple Referencescited by 48
  16. 2018Importance of Search and Evaluation Strategies in Neural Dialogue Modelingcited by 44
  17. 2019Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systemscited by 42
  18. 2018Response Generation by Context-aware Prototype Editingcited by 40
  19. 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
  20. 2020Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directionscited by 34
  21. 2021Reason first, then respond: Modular Generation for Knowledge-infused Dialoguecited by 32
  22. 2020Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systemscited by 26
  23. 2020Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluationcited by 22
  24. 2020Deploying Lifelong Open-Domain Dialogue Learningcited by 17
  25. 2018On Evaluating and Comparing Open Domain Dialog Systemscited by 16
  26. 2021A Survey of NLP-Related Crowdsourcing HITs: what works and what does notcited by 14
  27. 2020An Evaluation Protocol for Generative Conversational Systemscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 16; 2024: 19; 2025: 12; 2026: 3.

62 papers.

Sort:

By area:

  1. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  2. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  3. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  4. 2024LLM Critics Help Catch LLM Bugscited by 138
  5. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  6. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  7. 2022Evaluating Human-Language Model Interactioncited by 90
  8. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  9. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  10. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  11. 2022Keep Me Updated! Memory Management in Long-term Conversationscited by 62
  12. 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
38 more
  1. 2024Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theorycited by 60
  2. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  3. 2023Multi-Party Chat: Conversational Agents in Group Settings with Humans and Modelscited by 53
  4. 2023Diving Deep into Modes of Fact Hallucinations in Dialogue Systemscited by 32
  5. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  6. 2023Social Bias Probing: Fairness Benchmarking for Language Modelscited by 29
  7. 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
  8. 2024Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluationscited by 19
  9. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17
  10. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  11. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  12. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  13. 2023Mind the Gap Between Conversations for Improved Long-Term Dialogue Generationcited by 13
  14. 2025MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistancecited by 13
  15. 2022Towards Boosting the Open-Domain Chatbot with Human Feedbackcited by 11
  16. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  17. 2023Improving Open Language Models by Learning from Organic Interactionscited by 10
  18. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluationcited by 10
  19. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  20. 2022Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systemscited by 9
  21. 2022Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systemscited by 9
  22. 2025Evaluation and Facilitation of Online Discussions in the LLM Era: A Surveycited by 8
  23. 2024Multi-Document Grounded Multi-Turn Synthetic Dialog Generationcited by 7
  24. 2024Mixed-Session Conversation with Egocentric Memorycited by 7
  25. 2024PersonalityChat: Conversation Distillation for Personalized Dialog Modeling with Facts and Traitscited by 6
  26. 2022FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flowscited by 4
  27. 2023Psychological Metrics for Dialog System Evaluationcited by 4
  28. 2023Dialogue Quality and Emotion Annotations for Customer Support Conversationscited by 4
  29. 2024Assessing The Potential Of Mid-Sized Language Models For Clinical QAcited by 4
  30. 2025Are Bias Evaluation Methods Biased ?cited by 4
  31. 2022PoE: a Panel of Experts for Generalized Automatic Dialogue Assessmentcited by 3
  32. 2024ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmarkcited by 3
  33. 2025Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarkscited by 3
  34. 2025Modeling the One-to-Many Property in Open-Domain Dialogue with LLMscited by 3
  35. 2024An Interpretable and Crosslingual Method for Evaluating Second-Language Dialoguescited by 2
  36. 2025Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversationscited by 2
  37. 2023The HCI Aspects of Public Deployment of Research Chatbots: A User Study, Design Recommendations, and Open Challengescited by 1
  38. 2023Rating Prediction in Conversational Task Assistants with Behavioral and Conversational-Flow Featurescited by 1

50 most cited of 62 All 62 (JSON)