mlsanityData to 14 Oct 2026

Cooperative Exploration for Multi-Agent Deep Reinforcement Learning

arXiv 2107.11444 ↗First posted 23 Jul 2021cs.AIcited by 57

Not on the map. Why?

Abstract (arXiv)

Exploration is critical for good results in deep reinforcement learning and has attracted much attention. However, existing multi-agent deep reinforcement learning algorithms still use mostly noise-based techniques. Very recently, exploration methods that consider cooperation among multiple agents have been developed. However, existing methods suffer from a common challenge: agents struggle to identify states that are worth exploring, and hardly coordinate exploration efforts toward those states. To address this shortcoming, in this paper, we propose cooperative multi-agent exploration (CMAE): agents share a common goal while exploring. The goal is selected from multiple projected state spaces via a normalized entropy-based technique. Then, agents are trained to reach this goal in a coordinated manner. We demonstrate that CMAE consistently outperforms baselines on various tasks, including a sparse-reward version of the multiple-particle environment (MPE) and the Starcraft multi-agent challenge (SMAC).

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 48 of its 66 references that are in the map. First posted on arXiv 23 Jul 2021.

Builds on

48 of 66 references.

References · 48

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2017Hindsight Experience Replaycited by 1,136
  7. 2018Exploration by Random Network Distillationcited by 993
  8. 2017Deep Reinforcement Learning that Matterscited by 754
  9. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  10. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  12. 2019The StarCraft Multi-Agent Challengecited by 597
36 more
  1. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 575
  2. 2016Deep Exploration via Bootstrapped DQNcited by 527
  3. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  4. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  5. 2017Count-Based Exploration with Neural Density Modelscited by 257
  6. 2018Mean Field Multi-Agent Reinforcement Learningcited by 207
  7. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  8. 2019Efficient Exploration via State Marginal Matchingcited by 188
  9. 2019MAVEN: Multi-Agent Variational Explorationcited by 185
  10. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 184
  11. 2018Learning Attentional Communication for Multi-Agent Cooperationcited by 175
  12. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  13. 2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 148
  14. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 121
  15. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 117
  16. 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 116
  17. 2019Influence-Based Multi-Agent Explorationcited by 75
  18. 2018Modeling Others using Oneself in Multi-Agent Reinforcement Learningcited by 66
  19. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  20. 2020Learning Individually Inferred Communication for Multi-Agent Cooperationcited by 55
  21. 2020A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Taskscited by 52
  22. 2019PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learningcited by 47
  23. 2018Deep Reinforcement Learning for Swarm Systemscited by 45
  24. 2019Two Body Problem: Collaborative Visual Task Completioncited by 45
  25. 2017EX2: Exploration with Exemplar Models for Deep Reinforcement Learningcited by 42
  26. 2018GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithmscited by 40
  27. 2019Knowledge Flow: Improve Upon Your Teacherscited by 36
  28. 2020Succinct and Robust Multi-Agent Communication With Temporal Message Controlcited by 29
  29. 2020Optimistic Exploration even with a Pessimistic Initialisationcited by 25
  30. 2021GridToPix: Training Embodied Agents with Minimal Supervisioncited by 21
  31. 2018Coordinated Exploration in Concurrent Reinforcement Learningcited by 17
  32. 2019Scaled Autonomy: Enabling Human Operators to Control Robot Fleetscited by 16
  33. 2020Model-based Policy Optimization with Unsupervised Model Adaptationcited by 16
  34. 2021Neurosymbolic Transformers for Multi-Agent Communicationcited by 13
  35. 2020High-Throughput Synchronous Deep RLcited by 9
  36. 2019Concurrent Meta Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 13; 2023: 14; 2024: 10; 2025: 11; 2026: 3.

57 papers.

Sort:

By area:

  1. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  2. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  3. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 49
  4. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  5. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 33
  6. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  7. 2023Pretrained Language Models as Visual Planners for Human Assistancecited by 28
  8. 2021The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigationcited by 27
  9. 2022Last-Mile Embodied Visual Navigationcited by 25
  10. 2021Learning Efficient Multi-Agent Cooperative Visual Explorationcited by 19
  11. 2021Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agentscited by 19
  12. 2021Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learningcited by 18
38 more
  1. 2022Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computationcited by 14
  2. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  3. 2024CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperationcited by 12
  4. 2022Near-Optimal Multi-Agent Learning for Safe Coverage Controlcited by 11
  5. 2022Asking for Knowledge: Training RL Agents to Query External Knowledge Using Languagecited by 10
  6. 2023FoX: Formation-aware exploration in multi-agent reinforcement learningcited by 9
  7. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9
  8. 2023Self-Motivated Multi-Agent Explorationcited by 7
  9. 2024Fast Peer Adaptation with Context-aware Explorationcited by 7
  10. 2023Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learningcited by 5
  11. 2024Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learningcited by 5
  12. 2023Optimistic Multi-Agent Policy Gradientcited by 4
  13. 2024Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learningcited by 4
  14. 2024The impact of behavioral diversity in multi-agent reinforcement learningcited by 4
  15. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4
  16. 2025Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibrationcited by 4
  17. 2023Iteratively Learn Diverse Strategies with State Distance Informationcited by 3
  18. 2024Multi-Agent Continuous Control with Generative Flow Networkscited by 3
  19. 2025Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Explorationcited by 3
  20. 2021Influence-Based Reinforcement Learning for Intrinsically-Motivated Agentscited by 2
  21. 2023Adaptive trajectory-constrained exploration strategy for deep reinforcement learningcited by 2
  22. 2024Meta-DiffuB: A Contextualized Sequence-to-Sequence Text Diffusion Model with Meta-Explorationcited by 2
  23. 2025Focusing Influence Mechanism for Multi-Agent Reinforcement Learningcited by 2
  24. 2022The StarCraft Multi-Agent Challenges+ : Learning of Multi-Stage Tasks and Environmental Factors without Precise Reward Functionscited by 1
  25. 2022A Policy Resonance Approach to Solve the Problem of Responsibility Diffusion in Multiagent Reinforcement Learningcited by 1
  26. 2022Hierarchical Strategies for Cooperative Multi-Agent Reinforcement Learningcited by 1
  27. 2023Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative Multi-Agent Reinforcement Learningcited by 1
  28. 2023Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learningcited by 1
  29. 2023Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learningcited by 1
  30. 2023Graph Exploration for Effective Multi-agent Q-Learningcited by 1
  31. 2023Never Explore Repeatedly in Multi-Agent Reinforcement Learningcited by 1
  32. 2025Topology Enhanced MARL for Multi-Agent Cooperative Decision-Making of CAVscited by 1
  33. 2025Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning via Incorporating Generalized Human Expertisecited by 1
  34. 2022Exploiting Semantic Epsilon Greedy Exploration Strategy in Multi-Agent Reinforcement Learningcited by 0
  35. 2022Classifying Ambiguous Identities in Hidden-Role Stochastic Games with Multi-Agent Reinforcement Learningcited by 0
  36. 2024MESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structurecited by 0
  37. 2024Mini Honor of Kings: A Lightweight Environment for Multi-Agent Reinforcement Learningcited by 0
  38. 2025Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learningcited by 0

50 most cited of 57 All 57 (JSON)