mlsanityData to 14 Oct 2026

Towards Standard Criteria for human evaluation of Chatbots: A Survey

arXiv 2105.11197 ↗First posted 24 May 2021cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Human evaluation is becoming a necessity to test the performance of Chatbots. However, off-the-shelf settings suffer the severe reliability and replication issues partly because of the extremely high diversity of criteria. It is high time to come up with standard criteria and exact definitions. To this end, we conduct a through investigation of 105 papers involving human evaluation for Chatbots. Deriving from this, we propose five standard criteria along with precise definitions.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 80 of its 126 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

80 of 126 references.

References · 80

  1. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  2. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  3. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  4. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  5. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  6. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  7. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  8. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  9. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  10. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  11. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  12. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
38 more
  1. 2020Knowledge-Grounded Dialogue Generation with Pre-trained Language Modelscited by 107
  2. 2017Adversarial Learning for Neural Dialogue Generationcited by 103
  3. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  4. 2020Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systemscited by 98
  5. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  6. 2019MoEL: Mixture of Empathetic Listenerscited by 91
  7. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  8. 2019Proactive Human-Machine Conversation with Explicit Conversation Goalscited by 88
  9. 2019Personalizing Dialogue Agents via Meta-Learningcited by 83
  10. 2020Interactive Path Reasoning on Graph for Conversational Recommendationcited by 72
  11. 2020Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structurecited by 60
  12. 2019Improving Multi-turn Dialogue Modelling with Utterance ReWritercited by 58
  13. 2018Towards Exploiting Background Knowledge for Building Conversation Systemscited by 57
  14. 2019Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddingscited by 52
  15. 2020Learning an Unreferenced Metric for Online Dialogue Evaluationcited by 52
  16. 2018Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memorycited by 49
  17. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
  18. 2018Importance of Search and Evaluation Strategies in Neural Dialogue Modelingcited by 44
  19. 2020CDL: Curriculum Dual Learning for Emotion-Controllable Response Generationcited by 43
  20. 2017Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Modelscited by 42
  21. 2019Recent Advances in Neural Question Generationcited by 42
  22. 2019Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systemscited by 42
  23. 2020Generate, Delete and Rewrite: A Three-Stage Framework for Improving Persona Consistency of Dialogue Generationcited by 41
  24. 2019Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluatorscited by 35
  25. 2019Incremental Transformer with Deliberation Decoder for Document Grounded Conversationscited by 35
  26. 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
  27. 2020Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweightcited by 34
  28. 2018Generating More Interesting Responses in Neural Conversation Models with Distributional Constraintscited by 33
  29. 2020Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-Start Userscited by 33
  30. 2018Augmenting Neural Response Generation with Context-Aware Topical Attentioncited by 27
  31. 2020Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systemscited by 26
  32. 2019Structuring Latent Spaces for Stylized Response Generationcited by 25
  33. 2020PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Musiccited by 24
  34. 2019Jointly Optimizing Diversity and Relevance in Neural Response Generationcited by 22
  35. 2019Entity-Consistent End-to-end Task-Oriented Dialogue System with KB Retrievercited by 21
  36. 2020Diversifying Dialogue Generation with Non-Conversational Textcited by 21
  37. 2016Multi-domain Neural Network Language Generation for Spoken Dialogue Systemscited by 18
  38. 2018An Auto-Encoder Matching Model for Learning Utterance-Level Semantic Dependency in Dialogue Generationcited by 17

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 0; 2025: 1.

6 papers.

  1. 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
  2. 2023Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sourcescited by 9
  3. 2023MedGPTEval: A Dataset and Benchmark to Evaluate Responses of Large Language Models in Medicinecited by 7
  4. 2023Hierarchical Evaluation Framework: Best Practices for Human Evaluationcited by 5
  5. 2025Communication is All You Need: Persuasion Dataset Construction via Multi-LLM Communicationcited by 5
  6. 2023PK-Chat: Pointer Network Guided Knowledge Driven Generative Dialogue Modelcited by 3