mlsanityData to 14 Oct 2026

A Survey of Exploration Methods in Reinforcement Learning

arXiv 2109.00157 ↗First posted 1 Sep 2021cs.LG cs.AIcited by 73

Not on the map. Why?

Cited by 24 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Exploration is an essential component of reinforcement learning algorithms, where agents need to learn how to predict and control unknown and often stochastic environments. Reinforcement learning agents depend crucially on exploration to obtain informative data for the learning process as the lack of enough information could hinder effective learning. In this article, we provide a survey of modern exploration methods in (Sequential) reinforcement learning, as well as a taxonomy of exploration methods.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Builds on 67 of its 216 references that are in the map. First posted on arXiv 1 Sep 2021.

Builds on

67 of 216 references.

References · 67

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  9. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  12. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
38 more
  1. 2019The StarCraft Multi-Agent Challengecited by 597
  2. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  3. 2016Deep Exploration via Bootstrapped DQNcited by 527
  4. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  5. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  6. 2018Is Q-learning Provably Efficient?cited by 487
  7. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  8. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  9. 2019A Theory of Regularized Markov Decision Processescited by 286
  10. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  11. 2017Noisy Networks for Explorationcited by 266
  12. 2017Count-Based Exploration with Neural Density Modelscited by 257
  13. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  14. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  15. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  16. 2017A unified view of entropy-regularized Markov decision processescited by 182
  17. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
  18. 2017Parameter Space Noise for Explorationcited by 161
  19. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  20. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 130
  21. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117
  22. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  23. 2018Count-Based Exploration with the Successor Representationcited by 112
  24. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  25. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  26. 2018Model-Based Active Explorationcited by 105
  27. 2017Meta Learning Shared Hierarchiescited by 93
  28. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  29. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  30. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
  31. 2017Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learningcited by 83
  32. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 68
  33. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  34. 2018Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learningcited by 66
  35. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  36. 2019Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDPcited by 55
  37. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 49
  38. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 44

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 17; 2023: 18; 2024: 19; 2025: 10; 2026: 8.

73 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  3. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  4. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 59
  5. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  6. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 29
  7. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28
  8. 2024Practice Makes Perfect: Planning to Learn Skill Parameter Policiescited by 23
  9. 2022Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performancecited by 22
  10. 2022General Intelligence Requires Rethinking Explorationcited by 20
  11. 2024Machine learning and information theory concepts towards an AI Mathematiciancited by 20
  12. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
38 more
  1. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  2. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
  3. 2022Towards Understanding How Machines Can Learn Causal Overhypothesescited by 14
  4. 2022Active Exploration for Robotic Manipulationcited by 10
  5. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  6. 2025MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Modelcited by 10
  7. 2022Accelerated Reinforcement Learning for Temporal Logic Control Objectivescited by 9
  8. 2023Maximum diffusion reinforcement learningcited by 9
  9. 2023AutoRL Hyperparameter Landscapescited by 8
  10. 2023Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Samplingcited by 8
  11. 2024Neural Network Approaches for Parameterized Optimal Controlcited by 8
  12. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  13. 2022Learning Causal Overhypotheses through Exploration in Children and Computational Modelscited by 7
  14. 2022Complex behavior from intrinsic motivation to occupy action-state path spacecited by 7
  15. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Biascited by 7
  16. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 6
  17. 2023MIMEx: Intrinsic Rewards from Masked Input Modelingcited by 6
  18. 2023Successor-Predecessor Intrinsic Explorationcited by 6
  19. 2024Random Latent Exploration for Deep Reinforcement Learningcited by 6
  20. 2024Identifying Policy Gradient Subspacescited by 5
  21. 2024Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimizationcited by 5
  22. 2024Sample-Efficient Reinforcement Learning with Temporal Logic Objectives: Leveraging the Task Specification to Guide Explorationcited by 5
  23. 2022Assessing Quality-Diversity Neuro-Evolution Algorithms Performance in Hard Exploration Problemscited by 4
  24. 2023Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learningcited by 4
  25. 2023PcLast: Discovering Plannable Continuous Latent Statescited by 4
  26. 2024Active Inference and Human--Computer Interactioncited by 4
  27. 2022Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Controlcited by 3
  28. 2022Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel Statescited by 3
  29. 2024R×R: Rapid eXploration for Reinforcement Learning via Sampling-based Reset Distributions and Imitation Pre-trainingcited by 3
  30. 2024Instance Temperature Knowledge Distillationcited by 3
  31. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
  32. 2025Exploring Large Action Sets with Hyperspherical Embeddings using von Mises-Fisher Samplingcited by 3
  33. 2026Flexible Empowerment at Reasoning with Extended Best-of-N Samplingcited by 3
  34. 2022The Role of Exploration for Task Transfer in Reinforcement Learningcited by 2
  35. 2024Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstractioncited by 2
  36. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RLcited by 2
  37. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  38. 2022Guided Exploration in Reinforcement Learning via Monte Carlo Critic Optimizationcited by 1

50 most cited of 73 All 73 (JSON)