# Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning

arXiv: https://arxiv.org/abs/1808.09442 · HTML: https://mlsanity.com/paper/1808.09442/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2018.

## Built on by (most cited first, 11 of 11)

- [Aligning Large Language Models with Human Preferences through Representation Engineering](https://mlsanity.com/paper/2312.15997/) (2023)
- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning](https://mlsanity.com/paper/1811.07550/) (2018)
- [DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialogues](https://mlsanity.com/paper/2103.01287/) (2021)
- [Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue](https://mlsanity.com/paper/2110.06905/) (2021)
- [High-Quality Diversification for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2106.00891/) (2021)
- [TREND: Trigger-Enhanced Relation-Extraction Network for Dialogues](https://mlsanity.com/paper/2108.13811/) (2021)
- ["Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs](https://mlsanity.com/paper/2204.11481/) (2022)
- [WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue](https://mlsanity.com/paper/2108.01487/) (2021)
- [Dual Hierarchical Dialogue Policy Learning for Legal Inquisitive Conversational Agents](https://mlsanity.com/paper/2605.14057/) (2026)

## Abstract (arXiv)

This paper presents a Discriminative Deep Dyna-Q (D3Q) approach to improving the effectiveness and robustness of Deep Dyna-Q (DDQ), a recently proposed framework that extends the Dyna-Q algorithm to integrate planning for task-completion dialogue policy learning. To obviate DDQ's high dependency on the quality of simulated experiences, we incorporate an RNN-based discriminator in D3Q to differentiate simulated experience from real user experience in order to control the quality of training data. Experiments show that D3Q significantly outperforms DDQ by controlling the quality of simulated experience used for planning. The effectiveness and robustness of D3Q is further demonstrated in a domain extension setting, where the agent's capability of adapting to a changing environment is tested.