mlsanityData to 14 Oct 2026

Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations

arXiv 2312.13871 ↗First posted 21 Dec 2023cs.CL cs.HCcited by 9

Not on the map. Why?

Abstract (arXiv)

This review gives an extensive overview of evaluation methods for task-oriented dialogue systems, paying special attention to practical applications of dialogue systems, for example for customer service. The review (1) provides an overview of the used constructs and metrics in previous work, (2) discusses challenges in the context of dialogue system evaluation and (3) develops a research agenda for the future of dialogue system evaluation. We conducted a systematic review of four databases (ACL, ACM, IEEE and Web of Science), which after screening resulted in 122 studies. Those studies were carefully analysed for the constructs and methods they proposed for evaluation. We found a wide variety in both constructs and methods. Especially the operationalisation is not always clearly reported. Newer developments concerning large language models are discussed in two contexts: to power dialogue systems and to use in the evaluation process. We hope that future work will take a more critical approach to the operationalisation and specification of the used constructs. To work towards this aim, this review ends with recommendations for evaluation and suggestions for outstanding questions.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 51 of its 205 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

51 of 205 references.

References · 51

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  7. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  8. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  9. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  10. 2020Evaluation of Text Generation: A Surveycited by 277
  11. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  12. 2017Why We Need New Evaluation Metrics for NLGcited by 233
38 more
  1. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 207
  2. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  3. 2020A Survey on Conversational Recommender Systemscited by 143
  4. 2017Key-Value Retrieval Networks for Task-Oriented Dialoguecited by 138
  5. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  6. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  7. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
  8. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  9. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  10. 2018BLEU is Not Suitable for the Evaluation of Text Simplificationcited by 78
  11. 2020Recent Advances and Challenges in Task-oriented Dialog Systemcited by 78
  12. 2023A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospectscited by 66
  13. 2019How should my chatbot interact? A survey on human-chatbot interaction designcited by 56
  14. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 53
  15. 2020Conceptual Metaphors Impact Perceptions of Human-AI Collaborationcited by 47
  16. 2021Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systemscited by 47
  17. 2020Targeting the Benchmark: On Methodology in Current Natural Language Processing Researchcited by 45
  18. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 43
  19. 2021Why AI is Harder Than We Thinkcited by 41
  20. 2020RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systemscited by 36
  21. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  22. 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
  23. 2022Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generationcited by 33
  24. 2022A Literature Survey of Recent Advances in Chatbotscited by 26
  25. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 26
  26. 2021Preregistering NLP Researchcited by 25
  27. 2021Translation Quality Assessment: A Brief Survey on Manual and Automatic Methodscited by 23
  28. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  29. 2021Towards Quantifiable Dialogue Coherence Evaluationcited by 21
  30. 2023Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLPcited by 21
  31. 2020Refine and Imitate: Reducing Repetition and Inconsistency in Persuasion Dialogues via Reinforcement Learning and Human Demonstrationcited by 20
  32. 2020Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluationcited by 19
  33. 2021Underreporting of errors in NLG output, and what to do about itcited by 19
  34. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
  35. 2021Software-Based Dialogue Systems: Survey, Taxonomy and Challengescited by 11
  36. 2023Conversational AI-Powered Design: ChatGPT as Designer, User, and Productcited by 8
  37. 2022A Generative User Simulator with GPT-based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systemscited by 6
  38. 2017DeepProbe: Information Directed Sequence Understanding and Chatbot Design via Recurrent Neural Networkscited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 7.

9 papers.

  1. 2024Rethinking the Alignment of Psychotherapy Dialogue Generation with Motivational Interviewing Strategiescited by 10
  2. 2024Script-Strategy Aligned Generation: Aligning LLMs with Expert-Crafted Dialogue Scripts and Therapeutic Strategies for Psychotherapycited by 7
  3. 2025The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogscited by 6
  4. 2025clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisationscited by 4
  5. 2025MADS: Multi-Agent Dialogue Simulation for Diverse Persuasion Data Generationcited by 2
  6. 2025ScheduleMe: Multi-Agent Calendar Assistantcited by 1
  7. 2025Dynamic Evaluation Framework for Personalized and Trustworthy Agents: A Multi-Session Approach to Preference Adaptabilitycited by 0
  8. 2025Beyond Static Evaluation: Rethinking the Assessment of Personalized Agent Adaptability in Information Retrievalcited by 0
  9. 2025Collaborative and Proactive Management of Task-Oriented Conversationscited by 0