mlsanityData to 14 Oct 2026

USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation

arXiv 2005.00456 ↗First posted 1 May 2020cs.CL cs.LGcited by 188

Not on the map. Why?

Abstract (arXiv)

The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for evaluating dialog models. To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog. USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog. USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, system-level: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally produces interpretable measures for several desirable properties of dialog.

Key facts. Cited by 188 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 30; 2022: 26; 2023: 40; 2024: 41; 2025: 22; 2026: 29.

188 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  4. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  5. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  7. 2023GPTScore: Evaluate as You Desirecited by 361
  8. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  9. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  10. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  11. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  12. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
38 more
  1. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  2. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 101
  3. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  4. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  5. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 78
  6. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  7. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 71
  8. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  9. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  10. 2024Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversationcited by 55
  11. 2023Calibrating LLM-Based Evaluatorcited by 53
  12. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  13. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  14. 2021Shades of BLEU, Flavours of Success: The Case of MultiWOZcited by 48
  15. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  16. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  17. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  18. 2022Deepfake Text Detection: Limitations and Opportunitiescited by 37
  19. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  20. 2024Are LLM-based Evaluators Confusing NLG Quality Criteria?cited by 36
  21. 2023Opportunities and Challenges in Neural Dialog Tutoringcited by 33
  22. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  23. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
  24. 2022Achieving Reliable Human Assessment of Open-Domain Dialogue Systemscited by 31
  25. 2023DialogBench: Evaluating LLMs as Human-like Dialogue Systemscited by 31
  26. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 31
  27. 2021Commonsense-Focused Dialogues for Response Generation: An Empirical Studycited by 30
  28. 2021Better than Average: Paired Evaluation of NLP Systemscited by 30
  29. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  30. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28
  31. 2023Asking Clarification Questions to Handle Ambiguity in Open-Domain QAcited by 26
  32. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgmentscited by 25
  33. 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
  34. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  35. 2025Goal Alignment in LLM-Based User Simulators for Conversational AIcited by 21
  36. 2024Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluationscited by 20
  37. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 20
  38. 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluationcited by 20

50 most cited of 188 All 188 (JSON)