mlsanityData to 7 Oct 2026

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

arXiv 1603.08023 ↗First posted 25 Mar 2016cs.CL cs.AI cs.LGcited by 356

Not on the map. Why?

Abstract (arXiv)

We investigate evaluation metrics for dialogue response generation systems where supervised labels, such as task completion, are not available. Recent works in response generation have adopted metrics from machine translation to compare a model's generated response to a single target response. We show that these metrics correlate very weakly with human judgements in the non-technical Twitter domain, and not at all in the technical Ubuntu domain. We provide quantitative and qualitative results highlighting specific weaknesses in existing metrics, and provide recommendations for future development of better automatic evaluation metrics for dialogue systems.

Key facts. Cited by 356 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 1; 2019: 0; 2020: 0; 2021: 88; 2022: 85; 2023: 78; 2024: 48; 2025: 33; 2026: 22.

356 papers.

Sort:

By area:

  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2022Red Teaming Language Models with Language Modelscited by 983
  3. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  4. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  5. 2021Towards Emotional Support Dialog Systemscited by 294
  6. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  7. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  8. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  9. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  10. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  11. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  12. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
38 more
  1. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  2. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  3. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  4. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  5. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  6. 2023Large Language Models for Automated Open-domain Scientific Hypotheses Discoverycited by 100
  7. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94
  8. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  9. 2022Evaluating Human-Language Model Interactioncited by 90
  10. 2021CEM: Commonsense-aware Empathetic Response Generationcited by 89
  11. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  12. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  13. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  14. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  15. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77
  16. 2021Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approachcited by 75
  17. 2021Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Featurescited by 74
  18. 2021Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systemscited by 73
  19. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  20. 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 66
  21. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  22. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  23. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  24. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  25. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  26. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  27. 2021A Plug-and-Play Method for Controlled Text Generationcited by 55
  28. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  29. 2021Shades of BLEU, Flavours of Success: The Case of MultiWOZcited by 48
  30. 2023Conversation Chronicles: Towards Diverse Temporal and Relational Dynamics in Multi-Session Conversationscited by 47
  31. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  32. 2024Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Modelscited by 44
  33. 2021CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generationcited by 42
  34. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  35. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  36. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  37. 2022PAL: Persona-Augmented Emotional Support Conversation Generationcited by 39
  38. 2024Synthetic Dialogue Dataset Generation using LLM Agentscited by 39

50 most cited of 356 All 356 (JSON)