mlsanityData to 7 Oct 2026

How to Evaluate Your Dialogue Models: A Review of Approaches

arXiv 2108.01369 ↗First posted 3 Aug 2021cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Evaluating the quality of a dialogue system is an understudied problem. The recent evolution of evaluation method motivated this survey, in which an explicit and comprehensive analysis of the existing methods is sought. We are first to divide the evaluation methods into three classes, i.e., automatic evaluation, human-involved evaluation and user simulator based evaluation. Then, each class is covered with main features and the related evaluation metrics. The existence of benchmarks, suitable for the evaluation of dialogue techniques are also discussed in detail. Finally, some open issues are pointed out to bring the evaluation method into a new frontier.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 47 of its 77 references that are in the map. First posted on arXiv 3 Aug 2021.

Builds on

47 of 77 references.

References · 47

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  5. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  6. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  7. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  8. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  9. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  10. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  11. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  12. 2016A Persona-Based Neural Conversation Modelcited by 221
35 more
  1. 2015A Neural Conversational Modelcited by 195
  2. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  3. 2016A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialoguescited by 174
  4. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  5. 2017ParlAI: A Dialog Research Software Platformcited by 139
  6. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  7. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  8. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  9. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  10. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  11. 2018Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systemscited by 70
  12. 2017Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generationcited by 61
  13. 2015Towards Universal Paraphrastic Sentence Embeddingscited by 59
  14. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  15. 2017End-to-End Task-Completion Neural Dialogue Systemscited by 52
  16. 2020ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systemscited by 52
  17. 2019Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple Referencescited by 48
  18. 2019Global-to-local Memory Pointer Networks for Task-Oriented Dialoguecited by 47
  19. 2018Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systemscited by 46
  20. 2019How to Build User Simulators to Train RL-based Dialog Systemscited by 44
  21. 2018Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systemscited by 42
  22. 2019Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systemscited by 42
  23. 2020Designing Precise and Robust Dialogue Response Evaluatorscited by 37
  24. 2018User Modeling for Task Oriented Dialoguescited by 36
  25. 2017Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learningcited by 31
  26. 2019ConvLab: Multi-Domain End-to-End Dialog System Platformcited by 29
  27. 2016On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systemscited by 28
  28. 2019Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responsescited by 22
  29. 2015deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targetscited by 21
  30. 2020Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluationcited by 19
  31. 2018Topic-based Evaluation for Conversational Botscited by 16
  32. 2017Adversarial Evaluation of Dialogue Modelscited by 12
  33. 2020Plato Dialogue System: A Flexible Conversational AI Research Platformcited by 9
  34. 2018Neural Multi-task Learning in Automated Assessmentcited by 6
  35. 2017Referenceless Quality Estimation for Natural Language Generationcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1.

2 papers.

  1. 2023User Simulation with Large Language Models for Evaluating Task-Oriented Dialoguecited by 19
  2. 2022Social Influence Dialogue Systems: A Survey of Datasets and Models For Social Influence Taskscited by 14