# Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning

arXiv: https://arxiv.org/abs/1801.06176 · HTML: https://mlsanity.com/paper/1801.06176/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 11 of its 40 references that are in the map.
- First posted on arXiv 18 Jan 2018.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 11 of 11)

- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1508.01745/) (2015)
- [The Predictron: End-To-End Learning and Planning](https://mlsanity.com/paper/1612.08810/) (2016)
- [A User Simulator for Task-Completion Dialogues](https://mlsanity.com/paper/1612.05688/) (2016)
- [Dialogue Learning With Human-In-The-Loop](https://mlsanity.com/paper/1611.09823/) (2016)
- [Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning](https://mlsanity.com/paper/1606.02560/) (2016)
- [On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems](https://mlsanity.com/paper/1605.07669/) (2016)
- [Continuously Learning Neural Dialogue Management](https://mlsanity.com/paper/1606.02689/) (2016)
- [Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1710.11277/) (2017)
- [Sub-domain Modelling for Dialogue Management with Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1706.06210/) (2017)
- [BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1608.05081/) (2016)

## Built on by (most cited first, 21 of 21)

- [Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning](https://mlsanity.com/paper/2305.13660/) (2023)
- [Controllable Mixed-Initiative Dialogue Generation through Prompting](https://mlsanity.com/paper/2305.04147/) (2023)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulation](https://mlsanity.com/paper/2402.05733/) (2024)
- [RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems](https://mlsanity.com/paper/2103.08057/) (2021)
- [Task-Oriented Dialogue System as Natural Language Generation](https://mlsanity.com/paper/2108.13679/) (2021)
- [Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue System](https://mlsanity.com/paper/2207.11762/) (2022)
- [Policy Gradients Incorporating the Future](https://mlsanity.com/paper/2108.02096/) (2021)
- [Integrating Pretrained Language Model for Dialogue Policy Learning](https://mlsanity.com/paper/2111.01398/) (2021)
- [Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filtering](https://mlsanity.com/paper/2205.10363/) (2022)
- [TREND: Trigger-Enhanced Relation-Extraction Network for Dialogues](https://mlsanity.com/paper/2108.13811/) (2021)
- ["Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs](https://mlsanity.com/paper/2204.11481/) (2022)
- [Hybrid Learning for Orchestrating Deep Learning Inference in Multi-user Edge-cloud Networks](https://mlsanity.com/paper/2202.11098/) (2022)
- [Scheduled Curiosity-Deep Dyna-Q: Efficient Exploration for Dialog Policy Learning](https://mlsanity.com/paper/2402.00085/) (2024)
- [A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots](https://mlsanity.com/paper/2111.01414/) (2021)
- [Who's in Charge? Roles and Responsibilities of Decision-Making Components in Conversational Robots](https://mlsanity.com/paper/2303.08470/) (2023)
- [Deep RL with Hierarchical Action Exploration for Dialogue Generation](https://mlsanity.com/paper/2303.13465/) (2023)
- [A Unified Framework for Slot based Response Generation in a Multimodal Dialogue System](https://mlsanity.com/paper/2305.17433/) (2023)
- [Contextual Policy Transfer in Reinforcement Learning Domains via Deep Mixtures-of-Experts](https://mlsanity.com/paper/2003.00203/) (2020)
- [Stock Trading Optimization through Model-based Reinforcement Learning with Resistance Support Relative Strength](https://mlsanity.com/paper/2205.15056/) (2022)
- [Why Guided Dialog Policy Learning performs well? Understanding the role of adversarial learning and its alternative](https://mlsanity.com/paper/2307.06721/) (2023)

## Abstract (arXiv)

Training a task-completion dialogue agent via reinforcement learning (RL) is costly because it requires many interactions with real users. One common alternative is to use a user simulator. However, a user simulator usually lacks the language complexity of human interlocutors and the biases in its design may tend to degrade the agent. To address these issues, we present Deep Dyna-Q, which to our knowledge is the first deep RL framework that integrates planning for task-completion dialogue policy learning. We incorporate into the dialogue agent a model of the environment, referred to as the world model, to mimic real user response and generate simulated experience. During dialogue policy learning, the world model is constantly updated with real user experience to approach real user behavior, and in turn, the dialogue agent is optimized using both real experience and simulated experience. The effectiveness of our approach is demonstrated on a movie-ticket booking task in both simulated and human-in-the-loop settings.