mlsanityData to 14 Oct 2026

Modeling Performance in Open-Domain Dialogue with PARADISE

arXiv 2110.11164 ↗First posted 21 Oct 2021cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

There has recently been an explosion of work on spoken dialogue systems, along with an increased interest in open-domain systems that engage in casual conversations on popular topics such as movies, books and music. These systems aim to socially engage, entertain, and even empathize with their users. Since the achievement of such social goals is hard to measure, recent research has used dialogue length or human ratings as evaluation metrics, and developed methods for automatically calculating novel metrics, such as coherence, consistency, relevance and engagement. Here we develop a PARADISE model for predicting the performance of Athena, a dialogue system that has participated in thousands of conversations with real users, while competing as a finalist in the Alexa Prize. We use both user ratings and dialogue length as metrics for dialogue quality, and experiment with predicting these metrics using automatic features that are both system dependent and independent. Our goal is to learn a general objective function that can be used to optimize the dialogue choices of any Alexa Prize system in real time and evaluate its performance. Our best model for predicting user ratings gets an R2 of .136 with a DistilBert model, and the best model for predicting length with system independent features gets an R2 of .865, suggesting that conversation length may be a more reliable measure for automatic training of dialogue systems.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 31 of its 69 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

31 of 69 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  6. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  7. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  8. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  9. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  10. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  11. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  12. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
19 more
  1. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  2. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  3. 2019Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systemscited by 36
  4. 2019Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluatorscited by 35
  5. 2015Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systemscited by 34
  6. 2019ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversationcited by 31
  7. 2020Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversationscited by 24
  8. 2019MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversationscited by 20
  9. 2020Policy-Driven Neural Response Generation for Knowledge-Grounded Dialogue Systemscited by 17
  10. 2018On Evaluating and Comparing Open Domain Dialog Systemscited by 16
  11. 2020Emora: An Inquisitive Social Chatbot Who Cares For Youcited by 16
  12. 2017End-to-end Conversation Modeling Track in DSTC6cited by 12
  13. 2018Sounding Board: A User-Centric and Content-Driven Social Chatbotcited by 11
  14. 2018Neural Response Ranking for Social Conversation: A Data-Efficient Approachcited by 9
  15. 2020Athena: Constructing Dialogues Dynamically with Discourse Constraintscited by 7
  16. 2018SlugNERDS: A Named Entity Recognition Tool for Open Domain Dialogue Systemscited by 5
  17. 2019Entertaining and Opinionated but Too Controlling: A Large-Scale User Study of an Open Domain Alexa Prize Systemcited by 5
  18. 2019SlugBot: Developing a Computational Model andFramework of a Novel Dialogue Genrecited by 3
  19. 2016Analysis of Temporal Features for Interaction Quality Estimationcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 2.

5 papers.

  1. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  2. 2024Role-Play Zero-Shot Prompting with Large Language Models for Open-Domain Human-Machine Conversationcited by 7
  3. 2023Improving Open-Domain Dialogue Evaluation with a Causal Inference Modelcited by 4
  4. 2023Let's Get Personal: Personal Questions Improve SocialBot Performance in the Alexa Prizecited by 0
  5. 2024Language Portability Strategies for Open-domain Dialogue with Pre-trained Language Models from High to Low Resource Languagescited by 0