mlsanityData to 7 Oct 2026

RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems

arXiv 1701.03079 ↗First posted 11 Jan 2017cs.CL cs.HC cs.IRcited by 88

Not on the map. Why?

Abstract (arXiv)

Open-domain human-computer conversation has been attracting increasing attention over the past few years. However, there does not exist a standard automatic evaluation metric for open-domain dialog systems; researchers usually resort to human annotation for model evaluation, which is time- and labor-intensive. In this paper, we propose RUBER, a Referenced metric and Unreferenced metric Blended Evaluation Routine, which evaluates a reply by taking into consideration both a groundtruth reply and a query (previous user-issued utterance). Our metric is learnable, but its training does not require labels of human satisfaction. Hence, RUBER is flexible and extensible to different datasets and languages. Experiments on both retrieval and generative dialog systems show that RUBER has a high correlation with human annotation.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jan 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 21; 2023: 17; 2024: 16; 2025: 4; 2026: 1.

88 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  3. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  4. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  5. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  6. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  7. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  8. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  9. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  10. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  11. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  12. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
38 more
  1. 2021Commonsense-Focused Dialogues for Response Generation: An Empirical Studycited by 30
  2. 2022CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generationcited by 29
  3. 2022Rethinking and Refining the Distinct Metriccited by 27
  4. 2022DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulationscited by 25
  5. 2021Towards Quantifiable Dialogue Coherence Evaluationcited by 21
  6. 2022StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoningcited by 21
  7. 2021Ditch the Gold Standard: Re-evaluating Conversational Question Answeringcited by 19
  8. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  9. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
  10. 2022What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluationcited by 18
  11. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  12. 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
  13. 2021Assessing Dialogue Systems with Distribution Distancescited by 16
  14. 2021GTM: A Generative Triple-Wise Model for Conversational Question Generationcited by 16
  15. 2021Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluationcited by 16
  16. 2021Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimizationcited by 15
  17. 2022Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulatorcited by 15
  18. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  19. 2024Does Negative Sampling Matter? A Review with Insights into its Theory and Applicationscited by 14
  20. 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
  21. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  22. 2022What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveyscited by 13
  23. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  24. 2024CriticEval: Evaluating Large Language Model as Criticcited by 13
  25. 2023U-NEED: A Fine-grained Dataset for User Needs-Centric E-commerce Conversational Recommendationcited by 12
  26. 2023Affective and Dynamic Beam Search for Story Generationcited by 12
  27. 2022An Equal-Size Hard EM Algorithm for Diverse Dialogue Generationcited by 11
  28. 2023Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Referencecited by 11
  29. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11
  30. 2023ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systemscited by 11
  31. 2024PairEval: Open-domain Dialogue Evaluation with Pairwise Comparisoncited by 11
  32. 2024SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluationcited by 11
  33. 2021DiSCoL: Toward Engaging Dialogue Systems through Conversational Line Guided Response Generationcited by 10
  34. 2021HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversationscited by 9
  35. 2022Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentationcited by 8
  36. 2021Language Model Augmented Relevance Scorecited by 7
  37. 2023RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialoguecited by 7
  38. 2021Meta-evaluation of Conversational Search Evaluation Metricscited by 6

50 most cited of 88 All 88 (JSON)