# Designing Precise and Robust Dialogue Response Evaluators

arXiv: https://arxiv.org/abs/2004.04908 · HTML: https://mlsanity.com/paper/2004.04908/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2020.

## Built on by (most cited first, 30 of 37)

- [Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering](https://mlsanity.com/paper/2104.08202/) (2021)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators](https://mlsanity.com/paper/2312.15407/) (2023)
- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [Automatic Evaluation and Moderation of Open-domain Dialogue Systems](https://mlsanity.com/paper/2111.02110/) (2021)
- [Empathic Conversations: A Multi-level Dataset of Contextualized Conversations](https://mlsanity.com/paper/2205.12698/) (2022)
- [Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation](https://mlsanity.com/paper/2308.16797/) (2023)
- [Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability](https://mlsanity.com/paper/2406.18365/) (2024)
- [Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2407.16714/) (2024)
- [Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach](https://mlsanity.com/paper/2102.10242/) (2021)
- [Assessing Dialogue Systems with Distribution Distances](https://mlsanity.com/paper/2105.02573/) (2021)
- [Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation](https://mlsanity.com/paper/2106.05894/) (2021)
- [Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization](https://mlsanity.com/paper/2109.06471/) (2021)
- [MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation](https://mlsanity.com/paper/2112.07194/) (2021)
- [xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark](https://mlsanity.com/paper/2310.08958/) (2023)
- [Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation](https://mlsanity.com/paper/2301.12004/) (2023)
- [PairEval: Open-domain Dialogue Evaluation with Pairwise Comparison](https://mlsanity.com/paper/2404.01015/) (2024)
- [Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4](https://mlsanity.com/paper/2306.12794/) (2023)
- [On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation](https://mlsanity.com/paper/2407.03841/) (2024)
- [Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information](https://mlsanity.com/paper/2305.16967/) (2023)
- [Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models](https://mlsanity.com/paper/2405.14646/) (2024)
- [MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialogue](https://mlsanity.com/paper/2206.09403/) (2022)
- [ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues](https://mlsanity.com/paper/2407.11660/) (2024)
- [Investigating the Impact of Pre-trained Language Models on Dialog Evaluation](https://mlsanity.com/paper/2110.01895/) (2021)
- [Psychological Metrics for Dialog System Evaluation](https://mlsanity.com/paper/2305.14757/) (2023)
- [Automatic Evaluation of Generative Models with Instruction Tuning](https://mlsanity.com/paper/2310.20072/) (2023)
- [OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMs](https://mlsanity.com/paper/2503.11858/) (2025)
- [PoE: a Panel of Experts for Generalized Automatic Dialogue Assessment](https://mlsanity.com/paper/2212.08992/) (2022)
- [DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimation](https://mlsanity.com/paper/2305.04720/) (2023)

## Abstract (arXiv)

Automatic dialogue response evaluator has been proposed as an alternative to automated metrics and human evaluation. However, existing automatic evaluators achieve only moderate correlation with human judgement and they are not robust. In this work, we propose to build a reference-free evaluator and exploit the power of semi-supervised training and pretrained (masked) language models. Experimental results demonstrate that the proposed evaluator achieves a strong correlation (> 0.6) with human judgement and generalizes robustly to diverse responses and corpora. We open-source the code and data in https://github.com/ZHAOTING/dialog-processing.