# Unsupervised Evaluation of Interactive Dialog with DialoGPT

arXiv: https://arxiv.org/abs/2006.12719 · HTML: https://mlsanity.com/paper/2006.12719/

## Key facts

- Cited by 139 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2020.

## Built on by (most cited first, 30 of 139)

- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment](https://mlsanity.com/paper/2402.14016/) (2024)
- [LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models](https://mlsanity.com/paper/2305.13711/) (2023)
- [Human-centric Dialog Training via Offline Reinforcement Learning](https://mlsanity.com/paper/2010.05848/) (2020)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study](https://mlsanity.com/paper/2304.00723/) (2023)
- [LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models](https://mlsanity.com/paper/2307.07889/) (2023)
- [Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation](https://mlsanity.com/paper/2109.06379/) (2021)
- [DynaEval: Unifying Turn and Dialogue Level Evaluation](https://mlsanity.com/paper/2106.01112/) (2021)
- [GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abuse](https://mlsanity.com/paper/2401.01523/) (2024)
- [Faithful Persona-based Conversational Dataset Generation with Large Language Models](https://mlsanity.com/paper/2312.10007/) (2023)
- [A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators](https://mlsanity.com/paper/2312.15407/) (2023)
- [Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances](https://mlsanity.com/paper/2106.02227/) (2021)
- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2511.00222/) (2025)
- [Leveraging Large Language Models for NLG Evaluation: Advances and Challenges](https://mlsanity.com/paper/2401.07103/) (2024)
- [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://mlsanity.com/paper/2407.07064/) (2024)
- [LMUnit: Fine-grained Evaluation with Natural Language Unit Tests](https://mlsanity.com/paper/2412.13091/) (2024)
- [Achieving Reliable Human Assessment of Open-Domain Dialogue Systems](https://mlsanity.com/paper/2203.05899/) (2022)
- [Commonsense-Focused Dialogues for Response Generation: An Empirical Study](https://mlsanity.com/paper/2109.06427/) (2021)
- [Automatic Evaluation and Moderation of Open-domain Dialogue Systems](https://mlsanity.com/paper/2111.02110/) (2021)
- [X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects](https://mlsanity.com/paper/2311.08788/) (2023)
- [Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward](https://mlsanity.com/paper/2504.03206/) (2025)
- [DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations](https://mlsanity.com/paper/2203.09711/) (2022)
- [Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation](https://mlsanity.com/paper/2308.16797/) (2023)
- [Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability](https://mlsanity.com/paper/2406.18365/) (2024)
- [What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation](https://mlsanity.com/paper/2203.13927/) (2022)
- [Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models](https://mlsanity.com/paper/2401.13298/) (2024)

## Abstract (arXiv)

It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research. Standard language generation metrics have been shown to be ineffective for dialog. This paper introduces the FED metric (fine-grained evaluation of dialog), an automatic evaluation metric which uses DialoGPT, without any fine-tuning or supervision. It also introduces the FED dataset which is constructed by annotating a set of human-system and human-human conversations with eighteen fine-grained dialog qualities. The FED metric (1) does not rely on a ground-truth response, (2) does not require training data and (3) measures fine-grained dialog qualities at both the turn and whole dialog levels. FED attains moderate to strong correlation with human judgement at both levels.