Data to 7 Oct 2026

Learning an Unreferenced Metric for Online Dialogue Evaluation

arXiv 2005.00583 ↗First posted 1 May 2020cs.CL cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

Evaluating the quality of a dialogue interaction between two agents is a difficult task, especially in open-domain chit-chat style dialogue. There have been recent efforts to develop automatic dialogue evaluation metrics, but most of them do not generalize to unseen datasets and/or need a human-generated reference response during inference, making it infeasible for online evaluation. Here, we propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances, and leverages the temporal transitions that exist between them. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 13; 2022: 18; 2023: 15; 2024: 4; 2025: 2.

52 papers.

Sort:

2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
2022Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterancescited by 32
2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
2022Why is constrained neural language generation particularly challenging?cited by 19
2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
2022What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluationcited by 18

38 more

2022Measuring and Improving Semantic Diversity of Dialogue Generationcited by 16
2021Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimizationcited by 15
2022Persona-Guided Planning for Controlling the Protagonist's Persona in Story Generationcited by 15
2023CoAScore: Chain-of-Aspects Prompting for NLG Evaluationcited by 14
2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
2022What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveyscited by 13
2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
2022An Equal-Size Hard EM Algorithm for Diverse Dialogue Generationcited by 11
2022Interactive Evaluation of Dialog Track at DSTC9cited by 9
2021Assessing Reference-Free Peer Evaluation for Machine Translationcited by 8
2023RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialoguecited by 7
2021Meta-evaluation of Conversational Search Evaluation Metricscited by 6
2021Towards Standard Criteria for human evaluation of Chatbots: A Surveycited by 6
2022Probing the Robustness of Trained Metrics for Conversational Dialogue Systemscited by 6
2023Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Informationcited by 6
2022MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialoguecited by 5
2022Evaluation of Question Answering Systems: Complexity of judging a natural languagecited by 5
2023A Survey of Personality, Persona, and Profile in Conversational Agents and Chatbotscited by 5
2021Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversationcited by 4
2023Improving Open-Domain Dialogue Evaluation with a Causal Inference Modelcited by 4
2023Automatic Evaluation of Generative Models with Instruction Tuningcited by 4
2023Dialogue Quality and Emotion Annotations for Customer Support Conversationscited by 4
2024Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversationscited by 4
2022Open-Domain Dialog Evaluation using Follow-Ups Likelihoodcited by 3
2022PoE: a Panel of Experts for Generalized Automatic Dialogue Assessmentcited by 3
2023DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimationcited by 3
2023Correction of Errors in Preference Ratings from Automated Metrics for Text Generationcited by 3
2021POSSCORE: A Simple Yet Effective Evaluation of Conversational Search with Part of Speech Labellingcited by 2
2023Three Ways of Using Large Language Models to Evaluate Chatcited by 2
2024Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translationcited by 2
2024Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluationcited by 2
2021Do Encoder Representations of Generative Dialogue Models Encode Sufficient Information about the Task ?cited by 1
2022CORAL: Contextual Response Retrievability Loss Function for Training Dialog Generation Modelscited by 1
2025BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generationcited by 1
2025Evaluating LLM-Generated Versus Human-Authored Responses in Role-Play Dialoguescited by 1
2021A Brief Study on the Effects of Training Generative Dialogue Models with a Semantic losscited by 0

50 most cited of 52 All 52 (JSON)