mlsanityData to 7 Oct 2026

Exploration in Deep Reinforcement Learning: A Survey

arXiv 2205.00824 ↗First posted 2 May 2022cs.LGcited by 158

Not on the map. Why?

Abstract (arXiv)

This paper reviews exploration techniques in deep reinforcement learning. Exploration techniques are of primary importance when solving sparse reward problems. In sparse reward problems, the reward is rare, which means that the agent will not find the reward often by acting randomly. In such a scenario, it is challenging for reinforcement learning to learn rewards and actions association. Thus more sophisticated exploration methods need to be devised. This review provides a comprehensive overview of existing exploration approaches, which are categorized based on the key contributions as follows reward novel states, reward diverse behaviours, goal-based methods, probabilistic methods, imitation-based methods, safe exploration and random-based methods. Then, the unsolved challenges are discussed to provide valuable future research directions. Finally, the approaches of different categories are compared in terms of complexity, computational effort and overall performance.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Builds on 84 of its 143 references that are in the map. First posted on arXiv 2 May 2022.

Builds on

84 of 143 references.

References · 84

  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  4. 2018Exploration by Random Network Distillationcited by 966
  5. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  7. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  8. 2016Deep Exploration via Bootstrapped DQNcited by 518
  9. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  11. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  12. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
38 more
  1. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  2. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  3. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  4. 2019Self-Supervised Exploration via Disagreementcited by 263
  5. 2017Noisy Networks for Explorationcited by 262
  6. 2017Count-Based Exploration with Neural Density Modelscited by 255
  7. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  8. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  9. 2016Learning to Navigate in Complex Environmentscited by 192
  10. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  11. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  12. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  13. 2017Parameter Space Noise for Explorationcited by 158
  14. 2017Deep Exploration via Randomized Value Functionscited by 155
  15. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
  16. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  17. 2018Self-Imitation Learningcited by 135
  18. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  19. 2019Behaviour Suite for Reinforcement Learningcited by 129
  20. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
  21. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  22. 2018Count-Based Exploration with the Successor Representationcited by 112
  23. 2018Episodic Curiosity through Reachabilitycited by 109
  24. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  25. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  26. 2018Model-Based Active Explorationcited by 105
  27. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  28. 2018Playing hard exploration games by watching YouTubecited by 98
  29. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  30. 2017Eigenoption Discovery through the Deep Successor Representationcited by 85
  31. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  32. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  33. 2019A survey on intrinsic motivation in reinforcement learningcited by 79
  34. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  35. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  36. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72
  37. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  38. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 23; 2024: 48; 2025: 57; 2026: 26.

158 papers.

Sort:

By area:

  1. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  2. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  3. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  4. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74
  5. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 63
  6. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 60
  7. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 58
  8. 2026HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundrycited by 37
  9. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Explorationcited by 33
  10. 2024Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Modelscited by 32
  11. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  12. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
38 more
  1. 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Rewardcited by 25
  2. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  3. 2022Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performancecited by 21
  4. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  5. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
  6. 2024A social path to human-like artificial intelligencecited by 20
  7. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  8. 2025CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Preventioncited by 17
  9. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  10. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVRcited by 16
  11. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstrationcited by 14
  12. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  13. 2023Thinker: Learning to Plan and Actcited by 13
  14. 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
  15. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 11
  16. 2025Learning Massively Multitask World Models for Continuous Controlcited by 11
  17. 2023AdaRefiner: Refining Decisions of Language Models with Adaptive Feedbackcited by 10
  18. 2024Toward Enhanced Reinforcement Learning-Based Resource Management via Digital Twin: Opportunities, Applications, and Challengescited by 10
  19. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  20. 2025MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Modelcited by 10
  21. 2024Monitored Markov Decision Processescited by 9
  22. 2024Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewardscited by 9
  23. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 9
  24. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 9
  25. 2022Generalization in Deep Reinforcement Learning for Robotic Navigation by Reward Shapingcited by 8
  26. 2023Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Samplingcited by 8
  27. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  28. 2025Writing as a testbed for open ended agentscited by 8
  29. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  30. 2025Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Drivingcited by 7
  31. 2023MIMEx: Intrinsic Rewards from Masked Input Modelingcited by 6
  32. 2024Models That Prove Their Own Correctnesscited by 6
  33. 2024World Models with Hints of Large Language Models for Goal Achievingcited by 6
  34. 2024Beyond Optimism: Exploration With Partially Observable Rewardscited by 6
  35. 2024Advanced Persistent Threats (APT) Attribution Using Deep Reinforcement Learningcited by 6
  36. 2025DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Surveycited by 6
  37. 2025Preference-Driven Multi-Objective Combinatorial Optimization with Conditional Computationcited by 6
  38. 2026How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1cited by 6

50 most cited of 158 All 158 (JSON)