mlsanityData to 7 Oct 2026

Designing Precise and Robust Dialogue Response Evaluators

arXiv 2004.04908 ↗First posted 10 Apr 2020cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Automatic dialogue response evaluator has been proposed as an alternative to automated metrics and human evaluation. However, existing automatic evaluators achieve only moderate correlation with human judgement and they are not robust. In this work, we propose to build a reference-free evaluator and exploit the power of semi-supervised training and pretrained (masked) language models. Experimental results demonstrate that the proposed evaluator achieves a strong correlation (> 0.6) with human judgement and generalizes robustly to diverse responses and corpora. We open-source the code and data in https://github.com/ZHAOTING/dialog-processing.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 10; 2022: 6; 2023: 12; 2024: 7; 2025: 2.

37 papers.

Sort:

  1. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  2. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  3. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  4. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  5. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  6. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  7. 2022Empathic Conversations: A Multi-level Dataset of Contextualized Conversationscited by 20
  8. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  9. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  10. 2024Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversationcited by 19
  11. 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
  12. 2021Assessing Dialogue Systems with Distribution Distancescited by 16
25 more
  1. 2021Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluationcited by 16
  2. 2021Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimizationcited by 15
  3. 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
  4. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  5. 2023Understanding the Effectiveness of Very Large Language Models on Dialog Evaluationcited by 12
  6. 2024PairEval: Open-domain Dialogue Evaluation with Pairwise Comparisoncited by 11
  7. 2023Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4cited by 10
  8. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluationcited by 10
  9. 2023Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Informationcited by 6
  10. 2022MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialoguecited by 5
  11. 2024ECoh: Turn-level Coherence Evaluation for Multilingual Dialoguescited by 5
  12. 2021Investigating the Impact of Pre-trained Language Models on Dialog Evaluationcited by 4
  13. 2023Psychological Metrics for Dialog System Evaluationcited by 4
  14. 2023Automatic Evaluation of Generative Models with Instruction Tuningcited by 4
  15. 2024Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Modelscited by 4
  16. 2025OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMscited by 4
  17. 2022PoE: a Panel of Experts for Generalized Automatic Dialogue Assessmentcited by 3
  18. 2023DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimationcited by 3
  19. 2023CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogscited by 3
  20. 2024ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmarkcited by 3
  21. 2021How to Evaluate Your Dialogue Models: A Review of Approachescited by 2
  22. 2022Generate, Evaluate, and Select: A Dialogue System with a Response Evaluator for Diversity-Aware Response Generationcited by 1
  23. 2023Towards Multilingual Automatic Dialogue Evaluationcited by 0
  24. 2023DiQAD: A Benchmark Dataset for End-to-End Open-domain Dialogue Assessmentcited by 0
  25. 2025Measuring the Robustness of Reference-Free Dialogue Evaluation Systemscited by 0