mlsanityData to 14 Oct 2026

Open-Domain Dialog Evaluation using Follow-Ups Likelihood

arXiv 2209.05185 ↗First posted 12 Sep 2022cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Automatic evaluation of open-domain dialogs remains an unsolved problem. Moreover, existing methods do not correlate strongly with human annotations. This paper presents a new automated evaluation method using follow-ups: we measure the probability that a language model will continue the conversation with a fixed set of follow-ups (e.g., not really relevant here, what are you trying to say). When compared against twelve existing methods, our new evaluation achieves the highest correlation with human evaluations.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

18 of 23 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  4. 2020Recipes for building an open-domain chatbotcited by 593
  5. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  6. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  7. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  8. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  9. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  10. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  11. 2020Learning an Unreferenced Metric for Online Dialogue Evaluationcited by 52
  12. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
6 more
  1. 2021Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterancescited by 40
  2. 2019Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systemscited by 36
  3. 2022DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulationscited by 25
  4. 2022What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluationcited by 18
  5. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  6. 2019Beyond Turing: Intelligent Agents Centered on the Usercited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024PairEval: Open-domain Dialogue Evaluation with Pairwise Comparisoncited by 11
  2. 2024Aligning Language Models Using Follow-up Likelihood as Reward Signalcited by 6
  3. 2023Post Turing: Mapping the landscape of LLM Evaluationcited by 4