mlsanityData to 7 Oct 2026

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

arXiv 2109.06668 ↗First posted 14 Sep 2021cs.AI cs.LG cs.MAcited by 57

Not on the map. Why?

Abstract (arXiv)

Deep Reinforcement Learning (DRL) and Deep Multi-agent Reinforcement Learning (MARL) have achieved significant successes across a wide range of domains, including game AI, autonomous vehicles, robotics, and so on. However, DRL and deep MARL agents are widely known to be sample inefficient that millions of interactions are usually needed even for relatively simple problem settings, thus preventing the wide application and deployment in real-industry scenarios. One bottleneck challenge behind is the well-known exploration problem, i.e., how efficiently exploring the environment and collecting informative experiences that could benefit policy learning towards the optimal ones. This problem becomes more challenging in complex environments with sparse rewards, noisy distractions, long horizons, and non-stationary co-learners. In this paper, we conduct a comprehensive survey on existing exploration methods for both single-agent and multi-agent RL. We start the survey by identifying several key challenges to efficient exploration. Beyond the above two main branches, we also include other notable exploration methods with different ideas and techniques. In addition to algorithmic analysis, we provide a comprehensive and unified empirical comparison of different exploration methods for DRL on a set of commonly used benchmarks. According to our algorithmic and empirical investigation, we finally summarize the open problems of exploration in DRL and deep MARL and point out a few future directions.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 159 of its 245 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

159 of 245 references.

References · 159

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2016OpenAI Gymcited by 1,684
  10. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  11. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
38 more
  1. 2017Hindsight Experience Replaycited by 1,096
  2. 2015Prioritized Experience Replaycited by 1,070
  3. 2018Exploration by Random Network Distillationcited by 966
  4. 2017Constrained Policy Optimizationcited by 928
  5. 2016Deep Variational Information Bottleneckcited by 915
  6. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  7. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  11. 2018A Tutorial on Bayesian Optimizationcited by 734
  12. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  13. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  14. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  15. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  16. 2019The StarCraft Multi-Agent Challengecited by 589
  17. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  18. 2016Deep Exploration via Bootstrapped DQNcited by 518
  19. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  20. 2018Is Q-learning Provably Efficient?cited by 485
  21. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  22. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  23. 2017A Tutorial on Thompson Samplingcited by 420
  24. 2018Reward Constrained Policy Optimizationcited by 397
  25. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  26. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  27. 2017Deep Reinforcement Learning: An Overviewcited by 327
  28. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  29. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  30. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  31. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  32. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  33. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  34. 2019Self-Supervised Exploration via Disagreementcited by 263
  35. 2016Variational Intrinsic Controlcited by 262
  36. 2017Noisy Networks for Explorationcited by 262
  37. 2018Distributed Prioritized Experience Replaycited by 260
  38. 2020First return, then explorecited by 256

50 most cited of 159 All 159 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 11; 2024: 16; 2025: 21; 2026: 3.

57 papers.

Sort:

By area:

  1. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74
  2. 2023Small batch deep reinforcement learningcited by 35
  3. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
  4. 2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
  5. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 23
  6. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  7. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
  8. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replaycited by 19
  9. 2025Multi-Agent Reinforcement Learning for Resources Allocation Optimization: A Surveycited by 14
  10. 2024VL-TGS: Trajectory Generation and Selection using Vision Language Models in Mapless Outdoor Environmentscited by 11
  11. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  12. 2022EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Modelcited by 8
38 more
  1. 2023Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learningcited by 8
  2. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  3. 2023OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environmentscited by 7
  4. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  5. 2025Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Drivingcited by 7
  6. 2023Robust Multi-Agent Reinforcement Learning by Mutual Information Regularizationcited by 6
  7. 2024The Exploration-Exploitation Dilemma Revisited: An Entropy Perspectivecited by 5
  8. 2021Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learningcited by 4
  9. 2024Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learningcited by 4
  10. 2024Guiding Multi-agent Multi-task Reinforcement Learning by a Hierarchical Framework with Logical Reward Shapingcited by 4
  11. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learningcited by 3
  12. 2025Leveraging Partial SMILES Validation Scheme for Enhanced Drug Design in Reinforcement Learning Frameworkscited by 3
  13. 2025Efficient Skill Discovery via Regret-Aware Optimizationcited by 3
  14. 2025Key Decision-Makers in Multi-Agent Debates: Who Holds the Power?cited by 3
  15. 2024Provably Efficient Information-Directed Sampling Algorithms for Multi-Agent Reinforcement Learningcited by 2
  16. 2025A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspectivecited by 2
  17. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RLcited by 2
  18. 2022Image Augmentation Based Momentum Memory Intrinsic Reward for Sparse Reward Visual Scenescited by 1
  19. 2022A Policy Resonance Approach to Solve the Problem of Responsibility Diffusion in Multiagent Reinforcement Learningcited by 1
  20. 2023Latent-Conditioned Policy Gradient for Multi-Objective Deep Reinforcement Learningcited by 1
  21. 2024State-Novelty Guided Action Persistence in Deep Reinforcement Learningcited by 1
  22. 2024Multi-agent Reinforcement Learning for Dynamic Dispatching in Material Handling Systemscited by 1
  23. 2024Multi-Agent Deep Q-Network with Layer-based Communication Channel for Autonomous Internal Logistics Vehicle Scheduling in Smart Manufacturingcited by 1
  24. 2025Cooperative Game-Theoretic Credit Assignment for Multi-Agent Policy Gradients via the Corecited by 1
  25. 2025Wasserstein Barycenter Soft Actor-Criticcited by 1
  26. 2025Topology Enhanced MARL for Multi-Agent Cooperative Decision-Making of CAVscited by 1
  27. 2025Diffusion Policies with Offline and Inverse Reinforcement Learning for Promoting Physical Activity in Older Adults Using Wearable Sensorscited by 1
  28. 2022Rollout Total Correlation for Deep Reinforcement Learningcited by 0
  29. 2022Progress and summary of reinforcement learning on energy management of MPS-EVcited by 0
  30. 2023SVDE: Scalable Value-Decomposition Exploration for Cooperative Multi-Agent Reinforcement Learningcited by 0
  31. 2024Emergent Braitenberg-style Behaviours for Navigating the ViZDoom `My Way Home' Labyrinthcited by 0
  32. 2024FPGA Divide-and-Conquer Placement using Deep Reinforcement Learningcited by 0
  33. 2024GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Modelcited by 0
  34. 2024Asynchronous Fractional Multi-Agent Deep Reinforcement Learning for Age-Minimal Mobile Edge Computingcited by 0
  35. 2024A Safety Modulator Actor-Critic Method in Model-Free Safe Reinforcement Learning and Application in UAV Hoveringcited by 0
  36. 2025Adventurer: Exploration with BiGAN for Deep Reinforcement Learningcited by 0
  37. 2025Inverse RL Scene Dynamics Learning for Nonlinear Predictive Control in Autonomous Vehiclescited by 0
  38. 2025Application of LLMs to Multi-Robot Path Planning and Task Allocationcited by 0

50 most cited of 57 All 57 (JSON)