mlsanityData to 7 Oct 2026

Unsupervised Evaluation of Interactive Dialog with DialoGPT

arXiv 2006.12719 ↗First posted 23 Jun 2020cs.CL cs.AI cs.HCcited by 139

Not on the map. Why?

Abstract (arXiv)

It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research. Standard language generation metrics have been shown to be ineffective for dialog. This paper introduces the FED metric (fine-grained evaluation of dialog), an automatic evaluation metric which uses DialoGPT, without any fine-tuning or supervision. It also introduces the FED dataset which is constructed by annotating a set of human-system and human-human conversations with eighteen fine-grained dialog qualities. The FED metric (1) does not rely on a ground-truth response, (2) does not require training data and (3) measures fine-grained dialog qualities at both the turn and whole dialog levels. FED attains moderate to strong correlation with human judgement at both levels.

Key facts. Cited by 139 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 19; 2022: 25; 2023: 37; 2024: 30; 2025: 18; 2026: 9.

139 papers.

Sort:

By area:

  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  2. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  3. 2023GPTScore: Evaluate as You Desirecited by 357
  4. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  5. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  6. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  7. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  8. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  9. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  10. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  11. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  12. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abusecited by 50
38 more
  1. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 45
  2. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  3. 2021Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterancescited by 40
  4. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  5. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  6. 2025Consistently Simulating Human Personas with Multi-Turn Reinforcement Learningcited by 35
  7. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  8. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  9. 2022Achieving Reliable Human Assessment of Open-Domain Dialogue Systemscited by 31
  10. 2021Commonsense-Focused Dialogues for Response Generation: An Empirical Studycited by 30
  11. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  12. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28
  13. 2022DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulationscited by 25
  14. 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Rewardcited by 25
  15. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  16. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  17. 2022What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluationcited by 18
  18. 2024DiaSynth: Synthetic Dialogue Generation Framework for Low Resource Dialogue Applicationscited by 18
  19. 2024Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Modelscited by 17
  20. 2025Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performancecited by 17
  21. 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
  22. 2023BatchEval: Towards Human-like Text Evaluationcited by 16
  23. 2021Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Modelscited by 15
  24. 2021Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimizationcited by 15
  25. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
  26. 2023Leveraging Large Language Models for Automated Dialogue Analysiscited by 14
  27. 2023CoAScore: Chain-of-Aspects Prompting for NLG Evaluationcited by 14
  28. 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
  29. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  30. 2022What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveyscited by 13
  31. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  32. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  33. 2021Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistencycited by 12
  34. 2023Understanding the Effectiveness of Very Large Language Models on Dialog Evaluationcited by 12
  35. 2021Exploring Dense Retrieval for Dialogue Response Selectioncited by 11
  36. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11
  37. 2023ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systemscited by 11
  38. 2024PairEval: Open-domain Dialogue Evaluation with Pairwise Comparisoncited by 11

50 most cited of 139 All 139 (JSON)