# Transferable Dialogue Systems and User Simulators

arXiv: https://arxiv.org/abs/2107.11904 · HTML: https://mlsanity.com/paper/2107.11904/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 25 Jul 2021.

## Builds on (most cited first, 30 of 30)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2009.12005/) (2020)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching](https://mlsanity.com/paper/2005.05298/) (2020)
- [A User Simulator for Task-Completion Dialogues](https://mlsanity.com/paper/1612.05688/) (2016)
- [Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models](https://mlsanity.com/paper/1902.08858/) (2019)
- [End-to-End Task-Completion Neural Dialogue Systems](https://mlsanity.com/paper/1703.01008/) (2017)
- [Paraphrase Augmented Task-Oriented Dialog Generation](https://mlsanity.com/paper/2004.07462/) (2020)
- [Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems](https://mlsanity.com/paper/1805.06966/) (2018)
- [Structured Fusion Networks for Dialog](https://mlsanity.com/paper/1907.10016/) (2019)
- [A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems](https://mlsanity.com/paper/1607.00070/) (2016)
- [Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access](https://mlsanity.com/paper/1609.00777/) (2016)
- [User Modeling for Task Oriented Dialogues](https://mlsanity.com/paper/1811.04369/) (2018)
- [ConvLab: Multi-Domain End-to-End Dialog System Platform](https://mlsanity.com/paper/1904.08637/) (2019)
- [Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models](https://mlsanity.com/paper/1709.06136/) (2017)
- [Multi-Domain Dialogue Acts and Response Co-Generation](https://mlsanity.com/paper/2004.12363/) (2020)
- [Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Models](https://mlsanity.com/paper/1910.03756/) (2019)
- [Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition](https://mlsanity.com/paper/2004.03809/) (2020)
- [End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning](https://mlsanity.com/paper/1711.10712/) (2017)
- [Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management](https://mlsanity.com/paper/1707.00130/) (2017)
- [An End-to-End Trainable Neural Network Model with Belief Tracking for Task-Oriented Dialog](https://mlsanity.com/paper/1708.05956/) (2017)
- [Collaborative Multi-Agent Dialogue Model Training Via Reinforcement Learning](https://mlsanity.com/paper/1907.05507/) (2019)
- [Teacher-Student Framework Enhanced Multi-domain Dialogue Generation](https://mlsanity.com/paper/1908.07137/) (2019)
- [Retrospective and Prospective Mixture-of-Generators for Task-oriented Dialogue Response Generation](https://mlsanity.com/paper/1911.08151/) (2019)

## Built on by (most cited first, 30 of 36)

- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection](https://mlsanity.com/paper/2111.14592/) (2021)
- [Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following](https://mlsanity.com/paper/2410.15553/) (2024)
- [Flipping the Dialogue: Training and Evaluating User Language Models](https://mlsanity.com/paper/2510.06552/) (2025)
- [RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning](https://mlsanity.com/paper/2409.14674/) (2024)
- [Controllable Dialogue Simulation with In-Context Learning](https://mlsanity.com/paper/2210.04185/) (2022)
- [SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation](https://mlsanity.com/paper/2209.06664/) (2022)
- [LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluation](https://mlsanity.com/paper/2305.13614/) (2023)
- [User Simulation for Evaluating Information Access Systems](https://mlsanity.com/paper/2306.08550/) (2023)
- [Think, Act, and Ask: Open-World Interactive Personalized Robot Navigation](https://mlsanity.com/paper/2310.07968/) (2023)
- [GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers](https://mlsanity.com/paper/2208.10817/) (2022)
- [A Survey on Recent Advances in Conversational Data Generation](https://mlsanity.com/paper/2405.13003/) (2024)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator](https://mlsanity.com/paper/2210.14529/) (2022)
- [Privacy-Preserving Domain Adaptation of Semantic Parsers](https://mlsanity.com/paper/2212.10520/) (2022)
- [Toward Self-learning End-to-End Task-Oriented Dialog Systems](https://mlsanity.com/paper/2201.06849/) (2022)
- [Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems](https://mlsanity.com/paper/2204.00763/) (2022)
- [Augmenting Multi-Turn Text-to-SQL Datasets with Self-Play](https://mlsanity.com/paper/2210.12096/) (2022)
- [KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning](https://mlsanity.com/paper/2211.16773/) (2022)
- [EmoUS: Simulating User Emotions in Task-Oriented Dialogues](https://mlsanity.com/paper/2306.01579/) (2023)
- [Addressing Resource and Privacy Constraints in Semantic Parsing Through Data Augmentation](https://mlsanity.com/paper/2205.08675/) (2022)
- [Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use](https://mlsanity.com/paper/2410.24218/) (2024)
- [A Generative User Simulator with GPT-based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systems](https://mlsanity.com/paper/2210.08692/) (2022)
- [Post-processing Networks: Method for Optimizing Pipeline Task-oriented Dialogue Systems using Reinforcement Learning](https://mlsanity.com/paper/2207.12185/) (2022)
- [DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite Automaton](https://mlsanity.com/paper/2402.04411/) (2024)
- [OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue System](https://mlsanity.com/paper/2312.16864/) (2023)
- [Dialogue Evaluation with Offline Reinforcement Learning](https://mlsanity.com/paper/2209.00876/) (2022)
- [Data Augmentation for Conversational AI](https://mlsanity.com/paper/2309.04739/) (2023)
- [Adversarial learning of neural user simulators for dialogue policy optimisation](https://mlsanity.com/paper/2306.00858/) (2023)
- [From Intents to Conversations: Generating Intent-Driven Dialogues with Contrastive Learning for Multi-Turn Classification](https://mlsanity.com/paper/2411.14252/) (2024)

## Abstract (arXiv)

One of the difficulties in training dialogue systems is the lack of training data. We explore the possibility of creating dialogue data through the interaction between a dialogue system and a user simulator. Our goal is to develop a modelling framework that can incorporate new dialogue scenarios through self-play between the two agents. In this framework, we first pre-train the two agents on a collection of source domain dialogues, which equips the agents to converse with each other via natural language. With further fine-tuning on a small amount of target domain data, the agents continue to interact with the aim of improving their behaviors using reinforcement learning with structured reward functions. In experiments on the MultiWOZ dataset, two practical transfer learning problems are investigated: 1) domain adaptation and 2) single-to-multiple domain transfer. We demonstrate that the proposed framework is highly effective in bootstrapping the performance of the two agents in transfer learning. We also show that our method leads to improvements in dialogue system performance on complete datasets.