# Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses

arXiv: https://arxiv.org/abs/1902.08832 · HTML: https://mlsanity.com/paper/1902.08832/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Feb 2019.

## Built on by (most cited first, 22 of 22)

- [OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics](https://mlsanity.com/paper/2105.08920/) (2021)
- [Perturbation CheckLists for Evaluating NLG Evaluation Metrics](https://mlsanity.com/paper/2109.05771/) (2021)
- [Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View](https://mlsanity.com/paper/2203.12258/) (2022)
- [Ditch the Gold Standard: Re-evaluating Conversational Question Answering](https://mlsanity.com/paper/2112.08812/) (2021)
- [CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation](https://mlsanity.com/paper/2409.02098/) (2024)
- [Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges](https://mlsanity.com/paper/2203.10012/) (2022)
- [Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons](https://mlsanity.com/paper/2203.06063/) (2022)
- [MDCure: A Scalable Pipeline for Multi-Document Instruction-Following](https://mlsanity.com/paper/2410.23463/) (2024)
- [Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems](https://mlsanity.com/paper/2212.09180/) (2022)
- [Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentation](https://mlsanity.com/paper/2106.02833/) (2021)
- [Probing the Robustness of Trained Metrics for Conversational Dialogue Systems](https://mlsanity.com/paper/2202.13887/) (2022)
- [The Critique of Critique](https://mlsanity.com/paper/2401.04518/) (2024)
- [An Adversarially-Learned Turing Test for Dialog Generation Models](https://mlsanity.com/paper/2104.08231/) (2021)
- [How to Evaluate Your Dialogue Models: A Review of Approaches](https://mlsanity.com/paper/2108.01369/) (2021)
- [Three Ways of Using Large Language Models to Evaluate Chat](https://mlsanity.com/paper/2308.06502/) (2023)
- [Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluation](https://mlsanity.com/paper/2406.01131/) (2024)
- [On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems](https://mlsanity.com/paper/2104.06335/) (2021)
- [CORAL: Contextual Response Retrievability Loss Function for Training Dialog Generation Models](https://mlsanity.com/paper/2205.10558/) (2022)
- [Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey](https://mlsanity.com/paper/2411.09166/) (2024)
- [On Text Style Transfer via Style Masked Language Models](https://mlsanity.com/paper/2210.06394/) (2022)
- [Dialogue You Can Trust: Human and AI Perspectives on Generated Conversations](https://mlsanity.com/paper/2409.01808/) (2024)
- [Measuring the Robustness of Reference-Free Dialogue Evaluation Systems](https://mlsanity.com/paper/2501.06728/) (2025)

## Abstract (arXiv)

Automatically evaluating the quality of dialogue responses for unstructured domains is a challenging problem. ADEM(Lowe et al. 2017) formulated the automatic evaluation of dialogue systems as a learning problem and showed that such a model was able to predict responses which correlate significantly with human judgements, both at utterance and system level. Their system was shown to have beaten word-overlap metrics such as BLEU with large margins. We start with the question of whether an adversary can game the ADEM model. We design a battery of targeted attacks at the neural network based ADEM evaluation system and show that automatic evaluation of dialogue systems still has a long way to go. ADEM can get confused with a variation as simple as reversing the word order in the text! We report experiments on several such adversarial scenarios that draw out counterintuitive scores on the dialogue responses. We take a systematic look at the scoring function proposed by ADEM and connect it to linear system theory to predict the shortcomings evident in the system. We also devise an attack that can fool such a system to rate a response generation system as favorable. Finally, we allude to future research directions of using the adversarial attacks to design a truly automated dialogue evaluation system.