# Bootstrapped Thompson Sampling and Deep Exploration

arXiv: https://arxiv.org/abs/1507.00300 · HTML: https://mlsanity.com/paper/1507.00300/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2015.

## Built on by (most cited first, 30 of 40)

- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Distinguishing the Knowable from the Unknowable with Language Models](https://mlsanity.com/paper/2402.03563/) (2024)
- [Approximate Thompson Sampling via Epistemic Neural Networks](https://mlsanity.com/paper/2302.09205/) (2023)
- [On the Importance of Exploration for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2306.05483/) (2023)
- [From Predictions to Decisions: The Importance of Joint Predictive Distributions](https://mlsanity.com/paper/2107.09224/) (2021)
- [The Neural Testbed: Evaluating Joint Predictions](https://mlsanity.com/paper/2110.04629/) (2021)
- [From Dirichlet to Rubin: Optimistic Exploration in RL without Bonuses](https://mlsanity.com/paper/2205.07704/) (2022)
- [Fine-Tuning Language Models via Epistemic Neural Networks](https://mlsanity.com/paper/2211.01568/) (2022)
- [Diffusion Approximations for Thompson Sampling in the Small Gap Regime](https://mlsanity.com/paper/2105.09232/) (2021)
- [Evaluating High-Order Predictive Distributions in Deep Learning](https://mlsanity.com/paper/2202.13509/) (2022)
- [Exploration-Driven Generative Interactive Environments](https://mlsanity.com/paper/2504.02515/) (2025)
- [Residual Bootstrap Exploration for Stochastic Linear Bandit](https://mlsanity.com/paper/2202.11474/) (2022)
- [Differential Good Arm Identification](https://mlsanity.com/paper/2303.07154/) (2023)
- [Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning](https://mlsanity.com/paper/2512.16911/) (2025)
- [Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent](https://mlsanity.com/paper/2402.10228/) (2024)
- [Contextual Thompson Sampling via Generation of Missing Data](https://mlsanity.com/paper/2502.07064/) (2025)
- [Risk Aware and Multi-Objective Decision Making with Distributional Monte Carlo Tree Search](https://mlsanity.com/paper/2102.00966/) (2021)
- [State-Aware Variational Thompson Sampling for Deep Q-Networks](https://mlsanity.com/paper/2102.03719/) (2021)
- [SOTIF Entropy: Online SOTIF Risk Quantification and Mitigation for Autonomous Driving](https://mlsanity.com/paper/2211.04009/) (2022)
- [Model-Free Active Exploration in Reinforcement Learning](https://mlsanity.com/paper/2407.00801/) (2024)
- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [From Optimality to Robustness: Dirichlet Sampling Strategies in Stochastic Bandits](https://mlsanity.com/paper/2111.09724/) (2021)
- [Addressing Missing Data Issue for Diffusion-based Recommendation](https://mlsanity.com/paper/2505.12283/) (2025)
- [Debiasing Samples from Online Learning Using Bootstrap](https://mlsanity.com/paper/2108.00236/) (2021)
- [Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise](https://mlsanity.com/paper/2203.01004/) (2022)
- [Multiplier Bootstrap-based Exploration](https://mlsanity.com/paper/2302.01543/) (2023)
- [State Representation Learning Using an Unbalanced Atlas](https://mlsanity.com/paper/2305.10267/) (2023)
- [CORe: Capitalizing On Rewards in Bandit Exploration](https://mlsanity.com/paper/2103.04387/) (2021)
- [The Implicit Delta Method](https://mlsanity.com/paper/2211.06457/) (2022)

## Abstract (arXiv)

This technical note presents a new approach to carrying out the kind of exploration achieved by Thompson sampling, but without explicitly maintaining or sampling from posterior distributions. The approach is based on a bootstrap technique that uses a combination of observed and artificially generated data. The latter serves to induce a prior distribution which, as we will demonstrate, is critical to effective exploration. We explain how the approach can be applied to multi-armed bandit and reinforcement learning problems and how it relates to Thompson sampling. The approach is particularly well-suited for contexts in which exploration is coupled with deep learning, since in these settings, maintaining or generating samples from a posterior distribution becomes computationally infeasible.