# Deep Exploration via Bootstrapped DQN

arXiv: https://arxiv.org/abs/1602.04621 · HTML: https://mlsanity.com/paper/1602.04621/

## Key facts

- Cited by 527 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2016.

## Built on by (most cited first, 30 of 527)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble](https://mlsanity.com/paper/2110.01548/) (2021)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble](https://mlsanity.com/paper/2107.00591/) (2021)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Supervised Pretraining Can Learn In-Context Reinforcement Learning](https://mlsanity.com/paper/2306.14892/) (2023)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2105.08140/) (2021)
- [Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning](https://mlsanity.com/paper/2202.11566/) (2022)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks](https://mlsanity.com/paper/2203.09168/) (2022)
- [To Believe or Not to Believe Your LLM](https://mlsanity.com/paper/2406.02543/) (2024)
- [Dropout Q-Functions for Doubly Efficient Reinforcement Learning](https://mlsanity.com/paper/2110.02034/) (2021)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research](https://mlsanity.com/paper/2109.13202/) (2021)
- [On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty](https://mlsanity.com/paper/2102.11409/) (2021)
- [Reward is enough for convex MDPs](https://mlsanity.com/paper/2106.00661/) (2021)
- [RORL: Robust Offline Reinforcement Learning via Conservative Smoothing](https://mlsanity.com/paper/2206.02829/) (2022)
- [Outcome-based Exploration for LLM Reasoning](https://mlsanity.com/paper/2509.06941/) (2025)
- [Simplifying Deep Temporal Difference Learning](https://mlsanity.com/paper/2407.04811/) (2024)
- [Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters](https://mlsanity.com/paper/2205.13703/) (2022)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective](https://mlsanity.com/paper/2105.05246/) (2021)
- [Generative Flow Networks as Entropy-Regularized RL](https://mlsanity.com/paper/2310.12934/) (2023)
- [Reward Uncertainty for Exploration in Preference-based Reinforcement Learning](https://mlsanity.com/paper/2205.12401/) (2022)

## Abstract (arXiv)

Efficient exploration in complex environments remains a major challenge for reinforcement learning. We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions. Unlike dithering strategies such as epsilon-greedy exploration, bootstrapped DQN carries out temporally-extended (or deep) exploration; this can lead to exponentially faster learning. We demonstrate these benefits in complex stochastic MDPs and in the large-scale Arcade Learning Environment. Bootstrapped DQN substantially improves learning times and performance across most Atari games.