# Open-Domain Dialog Evaluation using Follow-Ups Likelihood

arXiv: https://arxiv.org/abs/2209.05185 · HTML: https://mlsanity.com/paper/2209.05185/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 12 Sep 2022.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems](https://mlsanity.com/paper/2010.03994/) (2020)
- [DynaEval: Unifying Turn and Dialogue Level Evaluation](https://mlsanity.com/paper/2106.01112/) (2021)
- [Learning an Unreferenced Metric for Online Dialogue Evaluation](https://mlsanity.com/paper/2005.00583/) (2020)
- [Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems](https://mlsanity.com/paper/2011.00483/) (2020)
- [Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances](https://mlsanity.com/paper/2106.02227/) (2021)
- [Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems](https://mlsanity.com/paper/1911.01456/) (2019)
- [DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations](https://mlsanity.com/paper/2203.09711/) (2022)
- [What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation](https://mlsanity.com/paper/2203.13927/) (2022)
- [Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges](https://mlsanity.com/paper/2203.10012/) (2022)
- [Beyond Turing: Intelligent Agents Centered on the User](https://mlsanity.com/paper/1901.06613/) (2019)

## Built on by (most cited first, 3 of 3)

- [PairEval: Open-domain Dialogue Evaluation with Pairwise Comparison](https://mlsanity.com/paper/2404.01015/) (2024)
- [Aligning Language Models Using Follow-up Likelihood as Reward Signal](https://mlsanity.com/paper/2409.13948/) (2024)
- [Post Turing: Mapping the landscape of LLM Evaluation](https://mlsanity.com/paper/2311.02049/) (2023)

## Abstract (arXiv)

Automatic evaluation of open-domain dialogs remains an unsolved problem. Moreover, existing methods do not correlate strongly with human annotations. This paper presents a new automated evaluation method using follow-ups: we measure the probability that a language model will continue the conversation with a fixed set of follow-ups (e.g., not really relevant here, what are you trying to say). When compared against twelve existing methods, our new evaluation achieves the highest correlation with human evaluations.