# Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning

arXiv: https://arxiv.org/abs/1811.07550 · HTML: https://mlsanity.com/paper/1811.07550/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 19 Nov 2018.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 9 of 9)

- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1508.01745/) (2015)
- [A User Simulator for Task-Completion Dialogues](https://mlsanity.com/paper/1612.05688/) (2016)
- [End-to-End Task-Completion Neural Dialogue Systems](https://mlsanity.com/paper/1703.01008/) (2017)
- [Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access](https://mlsanity.com/paper/1609.00777/) (2016)
- [Continuously Learning Neural Dialogue Management](https://mlsanity.com/paper/1606.02689/) (2016)
- [Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning](https://mlsanity.com/paper/1808.09442/) (2018)
- [BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1608.05081/) (2016)

## Built on by (most cited first, 12 of 12)

- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2302.10342/) (2023)
- [Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agents](https://mlsanity.com/paper/2506.00320/) (2025)
- [Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue System](https://mlsanity.com/paper/2207.11762/) (2022)
- [Dyna-Mind: Learning to Simulate from Experience for Better AI Agents](https://mlsanity.com/paper/2510.09577/) (2025)
- [Integrating Pretrained Language Model for Dialogue Policy Learning](https://mlsanity.com/paper/2111.01398/) (2021)
- ["Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs](https://mlsanity.com/paper/2204.11481/) (2022)
- [Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue](https://mlsanity.com/paper/2406.14457/) (2024)
- [Scheduled Curiosity-Deep Dyna-Q: Efficient Exploration for Dialog Policy Learning](https://mlsanity.com/paper/2402.00085/) (2024)
- [ProEvent: An Event-centric Benchmark for Proactive Agents](https://mlsanity.com/paper/2607.17701/) (2026)
- [Dyna-AIL : Adversarial Imitation Learning by Planning](https://mlsanity.com/paper/1903.03234/) (2019)
- [A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots](https://mlsanity.com/paper/2111.01414/) (2021)

## Abstract (arXiv)

Training task-completion dialogue agents with reinforcement learning usually requires a large number of real user experiences. The Dyna-Q algorithm extends Q-learning by integrating a world model, and thus can effectively boost training efficiency using simulated experiences generated by the world model. The effectiveness of Dyna-Q, however, depends on the quality of the world model - or implicitly, the pre-specified ratio of real vs. simulated experiences used for Q-learning. To this end, we extend the recently proposed Deep Dyna-Q (DDQ) framework by integrating a switcher that automatically determines whether to use a real or simulated experience for Q-learning. Furthermore, we explore the use of active learning for improving sample efficiency, by encouraging the world model to generate simulated experiences in the state-action space where the agent has not (fully) explored. Our results show that by combining switcher and active learning, the new framework named as Switch-based Active Deep Dyna-Q (Switch-DDQ), leads to significant improvement over DDQ and Q-learning baselines in both simulation and human evaluations.