mlsanityData to 7 Oct 2026

Assessing Dialogue Systems with Distribution Distances

arXiv 2105.02573 ↗First posted 6 May 2021cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

An important aspect of developing dialogue systems is how to evaluate and compare the performance of different systems. Existing automatic evaluation metrics are based on turn-level quality evaluation and use average scores for system-level comparison. In this paper, we propose to measure the performance of a dialogue system by computing the distribution-wise distance between its generated conversations and real-world conversations. Specifically, two distribution-wise metrics, FBD and PRD, are developed and evaluated. Experiments on several dialogue corpora show that our proposed metrics correlate better with human judgments than existing metrics.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 21 of its 41 references that are in the map. First posted on arXiv 6 May 2021.

Builds on

21 of 41 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  9. 2018Assessing Generative Models via Precision and Recallcited by 401
  10. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  11. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  12. 2015A Neural Conversational Modelcited by 195
9 more
  1. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  2. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  3. 2017ParlAI: A Dialog Research Software Platformcited by 139
  4. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  5. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  6. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  7. 2015Towards Universal Paraphrastic Sentence Embeddingscited by 59
  8. 2019Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddingscited by 52
  9. 2020Designing Precise and Robust Dialogue Response Evaluatorscited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 4; 2023: 3; 2024: 2; 2025: 3; 2026: 1.

16 papers.

Sort:

By area:

  1. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  2. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  3. 2022On the Evaluation Metrics for Paraphrase Generationcited by 32
  4. 2021A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancycited by 21
  5. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  6. 2024A Multi-Armed Bandit Approach to Online Selection and Evaluation of Generative Modelscited by 13
  7. 2023Privately Customizing Prefinetuning to Better Match User Data in Federated Learningcited by 12
  8. 2022On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluationcited by 7
  9. 2021REAMsharp: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generationcited by 4
  10. 2023DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimationcited by 3
  11. 2025SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluationcited by 3
  12. 2022Mental Health Assessment for the Chatbotscited by 0
4 more
  1. 2024A Unifying Information-theoretic Perspective on Evaluating Generative Modelscited by 0
  2. 2025Optimus: A Robust Defense Framework for Mitigating Toxicity while Fine-Tuning Conversational AIcited by 0
  3. 2025SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluationcited by 0
  4. 2026Coherence-Aware Distributional Evaluation of Open-Ended Text Generationcited by 0