# Modeling Performance in Open-Domain Dialogue with PARADISE arXiv: https://arxiv.org/abs/2110.11164 ยท HTML: https://mlsanity.com/paper/2110.11164/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 69 references that are in the map. - First posted on arXiv 21 Oct 2021. ## Builds on (most cited first, 30 of 31) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019) - [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018) - [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017) - [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016) - [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016) - [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015) - [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020) - [The Design and Implementation of XiaoIce, an Empathetic Social Chatbot](https://mlsanity.com/paper/1812.08989/) (2018) - [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020) - [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021) - [Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems](https://mlsanity.com/paper/1911.01456/) (2019) - [Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators](https://mlsanity.com/paper/1904.13015/) (2019) - [Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems](https://mlsanity.com/paper/1511.06931/) (2015) - [ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversation](https://mlsanity.com/paper/1910.12129/) (2019) - [Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations](https://mlsanity.com/paper/2008.12348/) (2020) - [MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversations](https://mlsanity.com/paper/1908.10023/) (2019) - [Policy-Driven Neural Response Generation for Knowledge-Grounded Dialogue Systems](https://mlsanity.com/paper/2005.12529/) (2020) - [On Evaluating and Comparing Open Domain Dialog Systems](https://mlsanity.com/paper/1801.03625/) (2018) - [Emora: An Inquisitive Social Chatbot Who Cares For You](https://mlsanity.com/paper/2009.04617/) (2020) - [End-to-end Conversation Modeling Track in DSTC6](https://mlsanity.com/paper/1706.07440/) (2017) - [Sounding Board: A User-Centric and Content-Driven Social Chatbot](https://mlsanity.com/paper/1804.10202/) (2018) - [Neural Response Ranking for Social Conversation: A Data-Efficient Approach](https://mlsanity.com/paper/1811.00967/) (2018) - [Athena: Constructing Dialogues Dynamically with Discourse Constraints](https://mlsanity.com/paper/2011.10683/) (2020) - [SlugNERDS: A Named Entity Recognition Tool for Open Domain Dialogue Systems](https://mlsanity.com/paper/1805.03784/) (2018) - [Entertaining and Opinionated but Too Controlling: A Large-Scale User Study of an Open Domain Alexa Prize System](https://mlsanity.com/paper/1908.04832/) (2019) - [SlugBot: Developing a Computational Model andFramework of a Novel Dialogue Genre](https://mlsanity.com/paper/1907.10658/) (2019) ## Built on by (most cited first, 5 of 5) - [Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges](https://mlsanity.com/paper/2203.10012/) (2022) - [Role-Play Zero-Shot Prompting with Large Language Models for Open-Domain Human-Machine Conversation](https://mlsanity.com/paper/2406.18460/) (2024) - [Improving Open-Domain Dialogue Evaluation with a Causal Inference Model](https://mlsanity.com/paper/2301.13372/) (2023) - [Let's Get Personal: Personal Questions Improve SocialBot Performance in the Alexa Prize](https://mlsanity.com/paper/2303.04953/) (2023) - [Language Portability Strategies for Open-domain Dialogue with Pre-trained Language Models from High to Low Resource Languages](https://mlsanity.com/paper/2407.01315/) (2024) ## Abstract (arXiv) There has recently been an explosion of work on spoken dialogue systems, along with an increased interest in open-domain systems that engage in casual conversations on popular topics such as movies, books and music. These systems aim to socially engage, entertain, and even empathize with their users. Since the achievement of such social goals is hard to measure, recent research has used dialogue length or human ratings as evaluation metrics, and developed methods for automatically calculating novel metrics, such as coherence, consistency, relevance and engagement. Here we develop a PARADISE model for predicting the performance of Athena, a dialogue system that has participated in thousands of conversations with real users, while competing as a finalist in the Alexa Prize. We use both user ratings and dialogue length as metrics for dialogue quality, and experiment with predicting these metrics using automatic features that are both system dependent and independent. Our goal is to learn a general objective function that can be used to optimize the dialogue choices of any Alexa Prize system in real time and evaluate its performance. Our best model for predicting user ratings gets an R2 of .136 with a DistilBert model, and the best model for predicting length with system independent features gets an R2 of .865, suggesting that conversation length may be a more reliable measure for automatic training of dialogue systems.