mlsanityData to 7 Oct 2026

Survey on Evaluation Methods for Dialogue Systems

arXiv 1905.04071 ↗First posted 10 May 2019cs.CL cs.AI cs.HCcited by 126

Not on the map. Why?

Abstract (arXiv)

In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost and time intensive. Thus, much work has been put into finding methods, which allow to reduce the involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented dialogue systems, conversational dialogue systems, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then by presenting the evaluation methods regarding this class.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 24; 2023: 26; 2024: 30; 2025: 14; 2026: 13.

126 papers.

Sort:

By area:

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  2. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  3. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  4. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  5. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  6. 2022Evaluating Human-Language Model Interactioncited by 90
  7. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  8. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  9. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  10. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  11. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  12. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
38 more
  1. 2023Are LLMs All You Need for Task-Oriented Dialogue?cited by 52
  2. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 50
  3. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  4. 2021Shades of BLEU, Flavours of Success: The Case of MultiWOZcited by 48
  5. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
  6. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  7. 2021Commonsense-Focused Dialogues for Response Generation: An Empirical Studycited by 30
  8. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  9. 2022Evaluating Mixed-initiative Conversational Search Systems via User Simulationcited by 24
  10. 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
  11. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 22
  12. 2024Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Modelscited by 19
  13. 2024Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluationscited by 19
  14. 2022What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluationcited by 18
  15. 2024IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answeringcited by 18
  16. 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
  17. 2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 16
  18. 2022Robust Dialogue State Tracking with Weak Supervision and Sparse Datacited by 15
  19. 2024DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Modelscited by 15
  20. 2022Empathetic Conversational Systems: A Review of Current Advances, Gaps, and Opportunitiescited by 14
  21. 2022A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialogcited by 14
  22. 2022Social Influence Dialogue Systems: A Survey of Datasets and Models For Social Influence Taskscited by 14
  23. 2021A Survey of Natural Language Generationcited by 13
  24. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  25. 2022What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveyscited by 13
  26. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  27. 2021LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcingcited by 11
  28. 2024Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systemscited by 11
  29. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  30. 2023Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4cited by 10
  31. 2025HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistantscited by 10
  32. 2022Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systemscited by 9
  33. 2023Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisationscited by 9
  34. 2023Schema-Guided User Satisfaction Modeling for Task-Oriented Dialoguescited by 8
  35. 2024AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMscited by 7
  36. 2024MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systemscited by 7
  37. 2025OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Modelscited by 7
  38. 2021Meta-evaluation of Conversational Search Evaluation Metricscited by 6

50 most cited of 126 All 126 (JSON)