# DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulation

arXiv: https://arxiv.org/abs/2512.20773 · HTML: https://mlsanity.com/paper/2512.20773/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Builds on (most cited first, 30 of 31)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data](https://mlsanity.com/paper/2404.14367/) (2024)
- [Offline RL for Natural Language Generation with Implicit Language Q Learning](https://mlsanity.com/paper/2206.11871/) (2022)
- [Adversarial Learning for Neural Dialogue Generation](https://mlsanity.com/paper/1701.06547/) (2017)
- [Multi-turn Reinforcement Learning from Preference Human Feedback](https://mlsanity.com/paper/2405.14655/) (2024)
- [Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning](https://mlsanity.com/paper/2305.13660/) (2023)
- [The Importance of Online Data: Understanding Preference Fine-tuning via Coverage](https://mlsanity.com/paper/2406.01462/) (2024)
- [Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems](https://mlsanity.com/paper/1805.06966/) (2018)
- [Controllable Dialogue Simulation with In-Context Learning](https://mlsanity.com/paper/2210.04185/) (2022)
- [A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems](https://mlsanity.com/paper/1607.00070/) (2016)
- [Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations](https://mlsanity.com/paper/2311.05584/) (2023)
- [Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models](https://mlsanity.com/paper/1709.06136/) (2017)
- [Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2402.13374/) (2024)
- [DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialogues](https://mlsanity.com/paper/2405.13028/) (2024)
- [Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition](https://mlsanity.com/paper/2004.03809/) (2020)
- [Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2106.08838/) (2021)
- [GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers](https://mlsanity.com/paper/2208.10817/) (2022)
- [ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format](https://mlsanity.com/paper/2211.17148/) (2022)
- [Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator](https://mlsanity.com/paper/2210.14529/) (2022)
- [Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations](https://mlsanity.com/paper/2411.05194/) (2024)
- [Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection](https://mlsanity.com/paper/2512.13240/) (2025)

## Built on by (most cited first, 5 of 5)

- [Fine-tuning LLMs for Passive Depression Severity Estimation from AI Mental Health Dialogue](https://mlsanity.com/paper/2606.17973/) (2026)
- [Dual-Agent Co-Training for Health Coaching via Implicit Adversarial Preference Optimization](https://mlsanity.com/paper/2605.07011/) (2026)
- [Investigating Assistant Bias in LLM User Simulators Using a Role Vector](https://mlsanity.com/paper/2609.00608/) (2026)
- [Optimizing Sparse Outcomes Through Dense Behavioral Signals via Value-Guided Preference Distillation](https://mlsanity.com/paper/2609.14648/) (2026)
- [Pretrained Persona Mixture Models and Tandem Models for Human Simulation](https://mlsanity.com/paper/2609.22607/) (2026)

## Abstract (arXiv)

Realistic user simulation is crucial for training and evaluating multi-turn dialogue systems, yet creating simulators that accurately replicate human behavior remains a significant challenge. An effective simulator must expose the failure modes of the systems under evaluation. This work introduces Direct Iterative Adversarial Learning (DIAL), an adversarial framework that iteratively enhances user simulator realism through a competitive dynamic between a generator (user simulator) and a discriminator. When applied to mental health support, a domain characterized by diverse failure types and a critical dependence on realistic user behavior for failure detection, DIAL restores lexical diversity diminished by supervised fine-tuning and drastically reduces discriminator accuracy. The resulting simulator exhibits a strong correlation between simulated and real failure occurrence rates while maintaining low distributional divergence of failure modes. These findings indicate that DIAL is a promising method for developing realistic user simulators in multi-turn dialogue, facilitating reliable and cost-effective system evaluation prior to deployment.