mlsanityData to 7 Oct 2026

Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration

arXiv 2509.10656 ↗First posted 12 Sep 2025cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

For groups of autonomous agents to achieve a particular goal, they must engage in coordination and long-horizon reasoning. Rather than relying on complex reward functions and explicit cooperation mechanisms, we ask what minimal ingredients are required for effective coordination and exploration to emerge in multi-agent settings. We investigate this question through self-supervised goal-reaching, where agents aim to maximize the likelihood of visiting a goal state rather than maximizing a reward. Despite a sparse feedback signal, we present empirical results that show self-supervised goal-reaching techniques enable agents to learn from such feedback. On MARL benchmarks, self-supervised goal-reaching outperforms alternative approaches that have access to the same sparse reward signal. Furthermore, we empirically demonstrate that multi-agent self-supervised goal-reaching approaches can be more robust than single-agent strategies. While there is no explicit exploration mechanism, this approach explores nontrivial intermediate coordination strategies in sparse settings where alternative approaches fail to achieve a single success.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

33 of 55 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  10. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  11. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  12. 2019The StarCraft Multi-Agent Challengecited by 589
21 more
  1. 2019Challenges of Real-World Reinforcement Learningcited by 282
  2. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  3. 2017Inverse Reward Designcited by 246
  4. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  5. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  6. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  7. 2019Goal-conditioned Imitation Learningcited by 159
  8. 2017Parameter Space Noise for Explorationcited by 158
  9. 2018Variational Option Discovery Algorithmscited by 121
  10. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  11. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 79
  12. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  13. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 59
  14. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  15. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  16. 2019Policy Continuation with Hindsight Inverse Dynamicscited by 22
  17. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  18. 2024Accelerating Goal-Conditioned RL Algorithms and Researchcited by 18
  19. 2019Reinforcement Learning without Ground-Truth Statecited by 15
  20. 2023FoX: Formation-aware exploration in multi-agent reinforcement learningcited by 9
  21. 2024LAGMA: LAtent Goal-guided Multi-Agent Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RLcited by 3
  2. 2026Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learningcited by 0
  3. 2026Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisationcited by 0