# A Survey of Exploration Methods in Reinforcement Learning

arXiv: https://arxiv.org/abs/2109.00157 · HTML: https://mlsanity.com/paper/2109.00157/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 216 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 30 of 67)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Minimax Regret Bounds for Reinforcement Learning](https://mlsanity.com/paper/1703.05449/) (2017)
- [Is Q-learning Provably Efficient?](https://mlsanity.com/paper/1807.03765/) (2018)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [A Theory of Regularized Markov Decision Processes](https://mlsanity.com/paper/1901.11275/) (2019)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Count-Based Exploration with Neural Density Models](https://mlsanity.com/paper/1703.01310/) (2017)
- [Provably Efficient Maximum Entropy Exploration](https://mlsanity.com/paper/1812.02690/) (2018)
- [(More) Efficient Reinforcement Learning via Posterior Sampling](https://mlsanity.com/paper/1306.0940/) (2013)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [A unified view of entropy-regularized Markov decision processes](https://mlsanity.com/paper/1705.07798/) (2017)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)

## Built on by (most cited first, 30 of 73)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models](https://mlsanity.com/paper/2508.10751/) (2025)
- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs](https://mlsanity.com/paper/2506.09026/) (2025)
- [CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models](https://mlsanity.com/paper/2509.09675/) (2025)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Practice Makes Perfect: Planning to Learn Skill Parameter Policies](https://mlsanity.com/paper/2402.15025/) (2024)
- [Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance](https://mlsanity.com/paper/2206.03787/) (2022)
- [General Intelligence Requires Rethinking Exploration](https://mlsanity.com/paper/2211.07819/) (2022)
- [Machine learning and information theory concepts towards an AI Mathematician](https://mlsanity.com/paper/2403.04571/) (2024)
- [The Benefits of Model-Based Generalization in Reinforcement Learning](https://mlsanity.com/paper/2211.02222/) (2022)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Diffusion Spectral Representation for Reinforcement Learning](https://mlsanity.com/paper/2406.16121/) (2024)
- [Towards Understanding How Machines Can Learn Causal Overhypotheses](https://mlsanity.com/paper/2206.08353/) (2022)
- [Active Exploration for Robotic Manipulation](https://mlsanity.com/paper/2210.12806/) (2022)
- [Exploring through Random Curiosity with General Value Functions](https://mlsanity.com/paper/2211.10282/) (2022)
- [MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model](https://mlsanity.com/paper/2510.11653/) (2025)
- [Accelerated Reinforcement Learning for Temporal Logic Control Objectives](https://mlsanity.com/paper/2205.04424/) (2022)
- [Maximum diffusion reinforcement learning](https://mlsanity.com/paper/2309.15293/) (2023)
- [AutoRL Hyperparameter Landscapes](https://mlsanity.com/paper/2304.02396/) (2023)
- [Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Sampling](https://mlsanity.com/paper/2312.11091/) (2023)
- [Neural Network Approaches for Parameterized Optimal Control](https://mlsanity.com/paper/2402.10033/) (2024)
- [The impact of intrinsic rewards on exploration in Reinforcement Learning](https://mlsanity.com/paper/2501.11533/) (2025)
- [Learning Causal Overhypotheses through Exploration in Children and Computational Models](https://mlsanity.com/paper/2202.10430/) (2022)
- [Complex behavior from intrinsic motivation to occupy action-state path space](https://mlsanity.com/paper/2205.10316/) (2022)
- [Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias](https://mlsanity.com/paper/2310.08558/) (2023)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [MIMEx: Intrinsic Rewards from Masked Input Modeling](https://mlsanity.com/paper/2305.08932/) (2023)
- [Successor-Predecessor Intrinsic Exploration](https://mlsanity.com/paper/2305.15277/) (2023)

## Abstract (arXiv)

Exploration is an essential component of reinforcement learning algorithms, where agents need to learn how to predict and control unknown and often stochastic environments. Reinforcement learning agents depend crucially on exploration to obtain informative data for the learning process as the lack of enough information could hinder effective learning. In this article, we provide a survey of modern exploration methods in (Sequential) reinforcement learning, as well as a taxonomy of exploration methods.