# How to Build User Simulators to Train RL-based Dialog Systems

arXiv: https://arxiv.org/abs/1909.01388 · HTML: https://mlsanity.com/paper/1909.01388/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2019.

## Built on by (most cited first, 30 of 46)

- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning](https://mlsanity.com/paper/2305.13660/) (2023)
- [Flipping the Dialogue: Training and Evaluating User Language Models](https://mlsanity.com/paper/2510.06552/) (2025)
- [Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles](https://mlsanity.com/paper/2502.18968/) (2025)
- [Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2402.13374/) (2024)
- [Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2106.08838/) (2021)
- [ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format](https://mlsanity.com/paper/2211.17148/) (2022)
- [Strength Lies in Differences! Improving Strategy Planning for Non-collaborative Dialogues via Diversified User Simulation](https://mlsanity.com/paper/2403.06769/) (2024)
- [Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing](https://mlsanity.com/paper/2305.08195/) (2023)
- [User Simulation with Large Language Models for Evaluating Task-Oriented Dialogue](https://mlsanity.com/paper/2309.13233/) (2023)
- [LAD: Language Models as Data for Zero-Shot Dialog](https://mlsanity.com/paper/2207.14393/) (2022)
- [Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator](https://mlsanity.com/paper/2210.14529/) (2022)
- [Social Influence Dialogue Systems: A Survey of Datasets and Models For Social Influence Tasks](https://mlsanity.com/paper/2210.05664/) (2022)
- [Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems](https://mlsanity.com/paper/2204.00763/) (2022)
- [KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning](https://mlsanity.com/paper/2211.16773/) (2022)
- [Human Choice Prediction in Language-based Persuasion Games: Simulation-based Off-Policy Evaluation](https://mlsanity.com/paper/2305.10361/) (2023)
- [Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue](https://mlsanity.com/paper/2110.06905/) (2021)
- [A Challenge on Semi-Supervised and Reinforced Task-Oriented Dialog Systems](https://mlsanity.com/paper/2207.02657/) (2022)
- [A Generative User Simulator with GPT-based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systems](https://mlsanity.com/paper/2210.08692/) (2022)
- [Simulating User Agents for Embodied Conversational-AI](https://mlsanity.com/paper/2410.23535/) (2024)
- [SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluation](https://mlsanity.com/paper/2510.11997/) (2025)
- [Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors](https://mlsanity.com/paper/2605.07847/) (2026)
- [Designing an Automatic Agent for Repeated Language based Persuasion Games](https://mlsanity.com/paper/2105.04976/) (2021)
- [Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants](https://mlsanity.com/paper/2410.12891/) (2024)
- [Ask, Clarify, Optimize: Human-LLM Agent Collaboration for Smarter Inventory Control](https://mlsanity.com/paper/2601.00121/) (2025)
- [Learning as Conversation: Dialogue Systems Reinforced for Information Acquisition](https://mlsanity.com/paper/2205.14748/) (2022)
- [High-Quality Diversification for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2106.00891/) (2021)
- [How to Evaluate Your Dialogue Models: A Review of Approaches](https://mlsanity.com/paper/2108.01369/) (2021)
- [Development of a Trust-Aware User Simulator for Statistical Proactive Dialog Modeling in Human-AI Teams](https://mlsanity.com/paper/2304.11913/) (2023)
- [METRO: Towards Strategy Induction from Expert Dialogue Transcripts for Non-collaborative Dialogues](https://mlsanity.com/paper/2604.11427/) (2026)

## Abstract (arXiv)

User simulators are essential for training reinforcement learning (RL) based dialog models. The performance of the simulator directly impacts the RL policy. However, building a good user simulator that models real user behaviors is challenging. We propose a method of standardizing user simulator building that can be used by the community to compare dialog system quality using the same set of user simulators fairly. We present implementations of six user simulators trained with different dialog planning and generation methods. We then calculate a set of automatic metrics to evaluate the quality of these simulators both directly and indirectly. We also ask human users to assess the simulators directly and indirectly by rating the simulated dialogs and interacting with the trained systems. This paper presents a comprehensive evaluation framework for user simulator study and provides a better understanding of the pros and cons of different user simulators, as well as their impacts on the trained systems.