mlsanityData to 13 Oct 2026

Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings

arXiv 1904.10635 ↗First posted 24 Apr 2019cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

Despite advances in open-domain dialogue systems, automatic evaluation of such systems is still a challenging problem. Traditional reference-based metrics such as BLEU are ineffective because there could be many valid responses for a given context that share no common words with reference responses. A recent work proposed Referenced metric and Unreferenced metric Blended Evaluation Routine (RUBER) to combine a learning-based metric, which predicts relatedness between a generated response and a given query, with reference-based metric; it showed high correlation with human judgments. In this paper, we explore using contextualized word embeddings to compute more accurate relatedness scores, thus better evaluation metrics. Experiments show that our evaluation metrics outperform RUBER, which is trained on static embeddings.

Key facts. Cited by 52 papers in the map (data through 13 Oct 2026). First posted on arXiv 24 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 18; 2022: 14; 2023: 12; 2024: 7; 2025: 1.

52 papers.

Sort:

By area:

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
  2. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  3. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  4. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  5. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  6. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  7. 2022CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generationcited by 29
  8. 2022DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulationscited by 25
  9. 2021Towards Quantifiable Dialogue Coherence Evaluationcited by 21
  10. 2022StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoningcited by 21
  11. 2024MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluationcited by 20
  12. 2021Ditch the Gold Standard: Re-evaluating Conversational Question Answeringcited by 19
38 more
  1. 2022What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluationcited by 18
  2. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  3. 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
  4. 2021Assessing Dialogue Systems with Distribution Distancescited by 16
  5. 2021Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluationcited by 16
  6. 2021Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimizationcited by 15
  7. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  8. 2024Does Negative Sampling Matter? A Review with Insights into its Theory and Applicationscited by 14
  9. 2021Knowledge Enhanced Fine-Tuning for Better Handling Unseen Entities in Dialogue Generationcited by 13
  10. 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
  11. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  12. 2022What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveyscited by 13
  13. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  14. 2023U-NEED: A Fine-grained Dataset for User Needs-Centric E-commerce Conversational Recommendationcited by 12
  15. 2021Exploring Dense Retrieval for Dialogue Response Selectioncited by 11
  16. 2023Event Causality Is Key to Computational Story Understandingcited by 11
  17. 2024PairEval: Open-domain Dialogue Evaluation with Pairwise Comparisoncited by 11
  18. 2021DiSCoL: Toward Engaging Dialogue Systems through Conversational Line Guided Response Generationcited by 10
  19. 2023RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialoguecited by 7
  20. 2021Meta-evaluation of Conversational Search Evaluation Metricscited by 6
  21. 2021Towards Standard Criteria for human evaluation of Chatbots: A Surveycited by 6
  22. 2022EnDex: Evaluation of Dialogue Engagingness at Scalecited by 6
  23. 2023Modeling What-to-ask and How-to-ask for Answer-unaware Conversational Question Generationcited by 6
  24. 2023Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Informationcited by 6
  25. 2022MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialoguecited by 5
  26. 2021REAMsharp: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generationcited by 4
  27. 2023Improving Open-Domain Dialogue Evaluation with a Causal Inference Modelcited by 4
  28. 2023Automatic Evaluation of Generative Models with Instruction Tuningcited by 4
  29. 2021User Response and Sentiment Prediction for Automatic Dialogue Evaluationcited by 3
  30. 2022PoE: a Panel of Experts for Generalized Automatic Dialogue Assessmentcited by 3
  31. 2023DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimationcited by 3
  32. 2021POSSCORE: A Simple Yet Effective Evaluation of Conversational Search with Part of Speech Labellingcited by 2
  33. 2022SelF-Eval: Self-supervised Fine-grained Dialogue Evaluationcited by 2
  34. 2024Emphasising Structured Information: Integrating Abstract Meaning Representation into LLMs for Enhanced Open-Domain Dialogue Evaluationcited by 2
  35. 2022Generate, Evaluate, and Select: A Dialogue System with a Response Evaluator for Diversity-Aware Response Generationcited by 1
  36. 2023Neural Conversation Models and How to Rein Them in: A Survey of Failures and Fixescited by 1
  37. 2025BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generationcited by 1
  38. 2022Relevance in Dialogue: Is Less More? An Empirical Comparison of Existing Metrics, and a Novel Simple Metriccited by 0

50 most cited of 52 All 52 (JSON)