# Deploying Lifelong Open-Domain Dialogue Learning

arXiv: https://arxiv.org/abs/2008.08076 · HTML: https://mlsanity.com/paper/2008.08076/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2020.

## Built on by (most cited first, 17 of 17)

- [Optimizing Prompts for Text-to-Image Generation](https://mlsanity.com/paper/2212.09611/) (2022)
- [Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling](https://mlsanity.com/paper/2107.03451/) (2021)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback](https://mlsanity.com/paper/2208.03270/) (2022)
- [Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk](https://mlsanity.com/paper/2401.05033/) (2024)
- [Reason first, then respond: Modular Generation for Knowledge-infused Dialogue](https://mlsanity.com/paper/2111.05204/) (2021)
- [When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels](https://mlsanity.com/paper/2210.15893/) (2022)
- [Inherently Explainable Reinforcement Learning in Natural Language](https://mlsanity.com/paper/2112.08907/) (2021)
- [Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls](https://mlsanity.com/paper/2208.03295/) (2022)
- [Towards Boosting the Open-Domain Chatbot with Human Feedback](https://mlsanity.com/paper/2208.14165/) (2022)
- [CHIRON: Rich Character Representations in Long-Form Narratives](https://mlsanity.com/paper/2406.10190/) (2024)
- [Targeted Data Acquisition for Evolving Negotiation Agents](https://mlsanity.com/paper/2106.07728/) (2021)
- [Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches and Future Directions](https://mlsanity.com/paper/2210.09894/) (2022)
- [How "open" are the conversations with open-domain chatbots? A proposal for Speech Event based evaluation](https://mlsanity.com/paper/2211.13560/) (2022)
- [CAST: Character-and-Scene Episodic Memory for Agents](https://mlsanity.com/paper/2602.06051/) (2026)
- [CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production](https://mlsanity.com/paper/2603.01973/) (2026)
- [Taming the Beast: Learning to Control Neural Conversational Models](https://mlsanity.com/paper/2108.10561/) (2021)

## Abstract (arXiv)

Much of NLP research has focused on crowdsourced static datasets and the supervised learning paradigm of training once and then evaluating test performance. As argued in de Vries et al. (2020), crowdsourced data has the issues of lack of naturalness and relevance to real-world use cases, while the static dataset paradigm does not allow for a model to learn from its experiences of using language (Silver et al., 2013). In contrast, one might hope for machine learning systems that become more useful as they interact with people. In this work, we build and deploy a role-playing game, whereby human players converse with learning agents situated in an open-domain fantasy world. We show that by training models on the conversations they have with humans in the game the models progressively improve, as measured by automatic metrics and online engagement scores. This learning is shown to be more efficient than crowdsourced data when applied to conversations with real users, as well as being far cheaper to collect.