# Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models

arXiv: https://arxiv.org/abs/1709.06136 · HTML: https://mlsanity.com/paper/1709.06136/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2017.

## Built on by (most cited first, 26 of 26)

- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Human-centric Dialog Training via Offline Reinforcement Learning](https://mlsanity.com/paper/2010.05848/) (2020)
- [Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning](https://mlsanity.com/paper/2305.13660/) (2023)
- [Neural Approaches to Conversational Information Retrieval](https://mlsanity.com/paper/2201.05176/) (2022)
- [Transferable Dialogue Systems and User Simulators](https://mlsanity.com/paper/2107.11904/) (2021)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator](https://mlsanity.com/paper/2308.11534/) (2023)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment](https://mlsanity.com/paper/2110.06536/) (2021)
- [IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022](https://mlsanity.com/paper/2205.13771/) (2022)
- [Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations](https://mlsanity.com/paper/2401.04883/) (2024)
- [Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions](https://mlsanity.com/paper/2305.10783/) (2023)
- [Toward Self-learning End-to-End Task-Oriented Dialog Systems](https://mlsanity.com/paper/2201.06849/) (2022)
- [A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions](https://mlsanity.com/paper/2504.16939/) (2025)
- [DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialogues](https://mlsanity.com/paper/2103.01287/) (2021)
- [A Generative User Simulator with GPT-based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systems](https://mlsanity.com/paper/2210.08692/) (2022)
- [DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulation](https://mlsanity.com/paper/2512.20773/) (2025)
- [Are Current Task-oriented Dialogue Systems Able to Satisfy Impolite Users?](https://mlsanity.com/paper/2210.12942/) (2022)
- [Joint System-Wise Optimization for Pipeline Goal-Oriented Dialog System](https://mlsanity.com/paper/2106.04835/) (2021)
- [IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents](https://mlsanity.com/paper/2407.08898/) (2024)
- [Every time I fire a conversational designer, the performance of the dialog system goes down](https://mlsanity.com/paper/2109.13029/) (2021)
- [Jointly Reinforced User Simulator and Task-oriented Dialog System with Simplified Generative Architecture](https://mlsanity.com/paper/2210.06706/) (2022)
- [JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy Learning](https://mlsanity.com/paper/2309.00230/) (2023)
- [Evaluating Task-Oriented Dialogue Consistency through Constraint Satisfaction](https://mlsanity.com/paper/2407.11857/) (2024)
- [Towards Detecting Inconsistencies in End-to-end Generated TODs](https://mlsanity.com/paper/2607.09338/) (2026)

## Abstract (arXiv)

In this paper, we present a deep reinforcement learning (RL) framework for iterative dialog policy optimization in end-to-end task-oriented dialog systems. Popular approaches in learning dialog policy with RL include letting a dialog agent to learn against a user simulator. Building a reliable user simulator, however, is not trivial, often as difficult as building a good dialog agent. We address this challenge by jointly optimizing the dialog agent and the user simulator with deep RL by simulating dialogs between the two agents. We first bootstrap a basic dialog agent and a basic user simulator by learning directly from dialog corpora with supervised training. We then improve them further by letting the two agents to conduct task-oriented dialogs and iteratively optimizing their policies with deep RL. Both the dialog agent and the user simulator are designed with neural network models that can be trained end-to-end. Our experiment results show that the proposed method leads to promising improvements on task success rate and total task reward comparing to supervised training and single-agent RL training baseline models.