# A Generative User Simulator with GPT-based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systems

arXiv: https://arxiv.org/abs/2210.08692 · HTML: https://mlsanity.com/paper/2210.08692/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](https://mlsanity.com/paper/2109.14739/) (2021)
- [UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2](https://mlsanity.com/paper/2012.03539/) (2020)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models](https://mlsanity.com/paper/1902.08858/) (2019)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [Shades of BLEU, Flavours of Success: The Case of MultiWOZ](https://mlsanity.com/paper/2106.05555/) (2021)
- [A Probabilistic End-To-End Task-Oriented Dialog Model with Latent Belief States towards Semi-Supervised Learning](https://mlsanity.com/paper/2009.08115/) (2020)
- [How to Build User Simulators to Train RL-based Dialog Systems](https://mlsanity.com/paper/1909.01388/) (2019)
- [Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems](https://mlsanity.com/paper/1805.06966/) (2018)
- [A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems](https://mlsanity.com/paper/1607.00070/) (2016)
- [User Modeling for Task Oriented Dialogues](https://mlsanity.com/paper/1811.04369/) (2018)
- [Transferable Dialogue Systems and User Simulators](https://mlsanity.com/paper/2107.11904/) (2021)
- [LAVA: Latent Action Spaces via Variational Auto-encoding for Dialogue Policy Optimization](https://mlsanity.com/paper/2011.09378/) (2020)
- [Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models](https://mlsanity.com/paper/1709.06136/) (2017)
- [Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition](https://mlsanity.com/paper/2004.03809/) (2020)
- [Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2106.08838/) (2021)
- [Collaborative Multi-Agent Dialogue Model Training Via Reinforcement Learning](https://mlsanity.com/paper/1907.05507/) (2019)
- [Building Markovian Generative Architectures over Pretrained LM Backbones for Efficient Task-Oriented Dialog Systems](https://mlsanity.com/paper/2204.06452/) (2022)

## Built on by (most cited first, 6 of 6)

- [Flipping the Dialogue: Training and Evaluating User Language Models](https://mlsanity.com/paper/2510.06552/) (2025)
- [Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk](https://mlsanity.com/paper/2401.05033/) (2024)
- [In-Context Learning User Simulators for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2306.00774/) (2023)
- [Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems](https://mlsanity.com/paper/2404.06762/) (2024)
- [Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations](https://mlsanity.com/paper/2312.13871/) (2023)
- [Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback](https://mlsanity.com/paper/2502.13298/) (2025)

## Abstract (arXiv)

Building user simulators (USs) for reinforcement learning (RL) of task-oriented dialog systems (DSs) has gained more and more attention, which, however, still faces several fundamental challenges. First, it is unclear whether we can leverage pretrained language models to design, for example, GPT-2 based USs, to catch up and interact with the recently advanced GPT-2 based DSs. Second, an important ingredient in a US is that the user goal can be effectively incorporated and tracked; but how to flexibly integrate goal state tracking and develop an end-to-end trainable US for multi-domains has remained to be a challenge. In this work, we propose a generative user simulator (GUS) with GPT-2 based architecture and goal state tracking towards addressing the above two challenges. Extensive experiments are conducted on MultiWOZ2.1. Different DSs are trained via RL with GUS, the classic agenda-based user simulator (ABUS) and other ablation simulators respectively, and are compared for cross-model evaluation, corpus-based evaluation and human evaluation. The GUS achieves superior results in all three evaluation tasks.