mlsanityData to 7 Oct 2026

Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning

arXiv 1801.06176 ↗First posted 18 Jan 2018cs.CL cs.AI cs.LGcited by 21

On the map: World models → World models for language agents

Abstract (arXiv)

Training a task-completion dialogue agent via reinforcement learning (RL) is costly because it requires many interactions with real users. One common alternative is to use a user simulator. However, a user simulator usually lacks the language complexity of human interlocutors and the biases in its design may tend to degrade the agent. To address these issues, we present Deep Dyna-Q, which to our knowledge is the first deep RL framework that integrates planning for task-completion dialogue policy learning. We incorporate into the dialogue agent a model of the environment, referred to as the world model, to mimic real user response and generate simulated experience. During dialogue policy learning, the world model is constantly updated with real user experience to approach real user behavior, and in turn, the dialogue agent is optimized using both real experience and simulated experience. The effectiveness of our approach is demonstrated on a movie-ticket booking task in both simulated and human-in-the-loop settings.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 11 of its 40 references that are in the map. First posted on arXiv 18 Jan 2018.

Builds on

11 of 40 references.

References · 11

  1. 2016Neural Belief Tracker: Data-Driven Dialogue State Trackingcited by 142
  2. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  3. 2016The Predictron: End-To-End Learning and Planningcited by 92
  4. 2016A User Simulator for Task-Completion Dialoguescited by 56
  5. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  6. 2016Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learningcited by 29
  7. 2016On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systemscited by 28
  8. 2016Continuously Learning Neural Dialogue Managementcited by 12
  9. 2017Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learningcited by 10
  10. 2017Sub-domain Modelling for Dialogue Management with Hierarchical Reinforcement Learningcited by 8
  11. 2016BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systemscited by 4

Built on by

Cited by year

Citing papers per year: 2020: 1; 2021: 6; 2022: 6; 2023: 6; 2024: 2.

21 papers.

Sort:

By area:

  1. 2023Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planningcited by 60
  2. 2023Controllable Mixed-Initiative Dialogue Generation through Promptingcited by 32
  3. 2022A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learningcited by 30
  4. 2024TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulationcited by 22
  5. 2021RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystemscited by 21
  6. 2021Task-Oriented Dialogue System as Natural Language Generationcited by 14
  7. 2022Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue Systemcited by 8
  8. 2021Policy Gradients Incorporating the Futurecited by 6
  9. 2021Integrating Pretrained Language Model for Dialogue Policy Learningcited by 5
  10. 2022Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filteringcited by 4
  11. 2021TREND: Trigger-Enhanced Relation-Extraction Network for Dialoguescited by 3
  12. 2022"Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogscited by 3
9 more
  1. 2022Hybrid Learning for Orchestrating Deep Learning Inference in Multi-user Edge-cloud Networkscited by 2
  2. 2024Scheduled Curiosity-Deep Dyna-Q: Efficient Exploration for Dialog Policy Learningcited by 2
  3. 2021A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrotscited by 1
  4. 2023Who's in Charge? Roles and Responsibilities of Decision-Making Components in Conversational Robotscited by 1
  5. 2023Deep RL with Hierarchical Action Exploration for Dialogue Generationcited by 1
  6. 2023A Unified Framework for Slot based Response Generation in a Multimodal Dialogue Systemcited by 1
  7. 2020Contextual Policy Transfer in Reinforcement Learning Domains via Deep Mixtures-of-Expertscited by 0
  8. 2022Stock Trading Optimization through Model-based Reinforcement Learning with Resistance Support Relative Strengthcited by 0
  9. 2023Why Guided Dialog Policy Learning performs well? Understanding the role of adversarial learning and its alternativecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2018Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning
  2. 2020Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
  3. 2022A Mixture-of-Expert Approach to RL-based Dialogue Management
  4. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents
  5. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  6. 2020Show Us the Way: Learning to Manage Dialog from Demonstrations
  7. 2020Learning Dialog Policies from Weak Demonstrations
  8. 2021Causal-aware Safe Policy Improvement for Task-oriented dialogue