# Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation

arXiv: https://arxiv.org/abs/2005.10716 · HTML: https://mlsanity.com/paper/2005.10716/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 22 of 22)

- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts](https://mlsanity.com/paper/2202.06523/) (2022)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings](https://mlsanity.com/paper/2404.16820/) (2024)
- [Threshold KNN-Shapley: A Linear-Time and Privacy-Friendly Approach to Data Valuation](https://mlsanity.com/paper/2308.15709/) (2023)
- [Label-Efficient Model Selection for Text Generation](https://mlsanity.com/paper/2402.07891/) (2024)
- [Efficient Data Shapley for Weighted Nearest Neighbor Algorithms](https://mlsanity.com/paper/2401.11103/) (2024)
- [GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering](https://mlsanity.com/paper/2104.10283/) (2021)
- [TokenShapley: Token Level Context Attribution with Shapley Value](https://mlsanity.com/paper/2507.05261/) (2025)
- [Robust Preference Optimization via Dynamic Target Margins](https://mlsanity.com/paper/2506.03690/) (2025)
- [DU-Shapley: A Shapley Value Proxy for Efficient Dataset Valuation](https://mlsanity.com/paper/2306.02071/) (2023)
- [Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons](https://mlsanity.com/paper/2203.06063/) (2022)
- [HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations](https://mlsanity.com/paper/2106.00162/) (2021)
- [Evaluating the Morphosyntactic Well-formedness of Generated Texts](https://mlsanity.com/paper/2103.16590/) (2021)
- [How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation](https://mlsanity.com/paper/2101.11298/) (2021)
- [Understanding and Improving the Exemplar-based Generation for Open-domain Conversation](https://mlsanity.com/paper/2112.06723/) (2021)
- [Learning the Legibility of Visual Text Perturbations](https://mlsanity.com/paper/2303.05077/) (2023)
- [What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts](https://mlsanity.com/paper/2111.00572/) (2021)
- [Capturing Human Cognitive Styles with Language: Towards an Experimental Evaluation Paradigm](https://mlsanity.com/paper/2502.13326/) (2025)
- [DiQAD: A Benchmark Dataset for End-to-End Open-domain Dialogue Assessment](https://mlsanity.com/paper/2310.16319/) (2023)
- [Reliable and Private Utility Signaling for Data Markets](https://mlsanity.com/paper/2511.07975/) (2025)

## Abstract (arXiv)

Open Domain dialog system evaluation is one of the most important challenges in dialog research. Existing automatic evaluation metrics, such as BLEU are mostly reference-based. They calculate the difference between the generated response and a limited number of available references. Likert-score based self-reported user rating is widely adopted by social conversational systems, such as Amazon Alexa Prize chatbots. However, self-reported user rating suffers from bias and variance among different users. To alleviate this problem, we formulate dialog evaluation as a comparison task. We also propose an automatic evaluation model CMADE (Comparison Model for Automatic Dialog Evaluation) that automatically cleans self-reported user ratings as it trains on them. Specifically, we first use a self-supervised method to learn better dialog feature representation, and then use KNN and Shapley to remove confusing samples. Our experiments show that CMADE achieves 89.2% accuracy in the dialog comparison task.