# BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems

arXiv: https://arxiv.org/abs/1711.05715 · HTML: https://mlsanity.com/paper/1711.05715/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2017.

## Built on by (most cited first, 30 of 30)

- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2105.08140/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [SGP-TOD: Building Task Bots Effortlessly via Schema-Guided LLM Prompting](https://mlsanity.com/paper/2305.09067/) (2023)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [Multi-facet Contextual Bandits: A Neural Network Perspective](https://mlsanity.com/paper/2106.03039/) (2021)
- [Learning Deformable Object Manipulation from Expert Demonstrations](https://mlsanity.com/paper/2207.10148/) (2022)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Bayesian Bellman Operators](https://mlsanity.com/paper/2106.05012/) (2021)
- [Dynamic Dialogue Policy for Continual Reinforcement Learning](https://mlsanity.com/paper/2204.05928/) (2022)
- [Toward Self-learning End-to-End Task-Oriented Dialog Systems](https://mlsanity.com/paper/2201.06849/) (2022)
- [Neural Collaborative Filtering Bandits via Meta Learning](https://mlsanity.com/paper/2201.13395/) (2022)
- [Fast online inference for nonlinear contextual bandit based on Generative Adversarial Network](https://mlsanity.com/paper/2202.08867/) (2022)
- [Meta Clustering of Neural Bandits](https://mlsanity.com/paper/2408.05586/) (2024)
- [Prescriptive Process Monitoring in Intelligent Process Automation with Chatbot Orchestration](https://mlsanity.com/paper/2212.06564/) (2022)
- [Enhancing End-to-End Multi-Task Dialogue Systems: A Study on Intrinsic Motivation Reinforcement Learning Algorithms for Improved Training and Adaptability](https://mlsanity.com/paper/2401.18040/) (2024)
- [TOD-Flow: Modeling the Structure of Task-Oriented Dialogues](https://mlsanity.com/paper/2312.04668/) (2023)
- [Deep Exploration with PAC-Bayes](https://mlsanity.com/paper/2402.03055/) (2024)
- [High-Quality Diversification for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2106.00891/) (2021)
- [What Does The User Want? Information Gain for Hierarchical Dialogue Policy Optimisation](https://mlsanity.com/paper/2109.07129/) (2021)
- [Bayesian Exploration Networks](https://mlsanity.com/paper/2308.13049/) (2023)
- [Analytically Tractable Bayesian Deep Q-Learning](https://mlsanity.com/paper/2106.11086/) (2021)
- [Knowledge-Guided Exploration in Deep Reinforcement Learning](https://mlsanity.com/paper/2210.15670/) (2022)
- [Learning How to Infer Partial MDPs for In-Context Adaptation and Exploration](https://mlsanity.com/paper/2302.04250/) (2023)
- [Scheduled Curiosity-Deep Dyna-Q: Efficient Exploration for Dialog Policy Learning](https://mlsanity.com/paper/2402.00085/) (2024)
- [Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization](https://mlsanity.com/paper/2305.03262/) (2023)
- [Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems](https://mlsanity.com/paper/2307.07255/) (2023)
- [Global Hierarchical Neural Networks using Hierarchical Softmax](https://mlsanity.com/paper/2308.01210/) (2023)
- [Recent Progress in Conversational AI](https://mlsanity.com/paper/2204.09719/) (2022)
- [Reinforcement Learning of Multi-Domain Dialog Policies Via Action Embeddings](https://mlsanity.com/paper/2207.00468/) (2022)

## Abstract (arXiv)

We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as ε-greedy, Boltzmann, bootstrapping, and intrinsic-reward-based ones. Additionally, we show that spiking the replay buffer with experiences from just a few successful episodes can make Q-learning feasible when it might otherwise fail.