# Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration

arXiv: https://arxiv.org/abs/2509.10656 · HTML: https://mlsanity.com/paper/2509.10656/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 12 Sep 2025.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [FACMAC: Factored Multi-Agent Centralised Policy Gradients](https://mlsanity.com/paper/2003.06709/) (2020)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [Variational Option Discovery Algorithms](https://mlsanity.com/paper/1807.10299/) (2018)
- [C-Learning: Learning to Achieve Goals via Recursive Classification](https://mlsanity.com/paper/2011.08909/) (2020)
- [Does Zero-Shot Reinforcement Learning Exist?](https://mlsanity.com/paper/2209.14935/) (2022)
- [Loss is its own Reward: Self-Supervision for Reinforcement Learning](https://mlsanity.com/paper/1612.07307/) (2016)
- [JaxMARL: Multi-Agent RL Environments and Algorithms in JAX](https://mlsanity.com/paper/2311.10090/) (2023)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer](https://mlsanity.com/paper/2206.10607/) (2022)
- [A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals](https://mlsanity.com/paper/2408.05804/) (2024)
- [Policy Continuation with Hindsight Inverse Dynamics](https://mlsanity.com/paper/1910.14055/) (2019)
- [Accelerating Goal-Conditioned RL Algorithms and Research](https://mlsanity.com/paper/2408.11052/) (2024)

## Built on by (most cited first, 3 of 3)

- [Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL](https://mlsanity.com/paper/2510.14129/) (2025)
- [Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.01865/) (2026)
- [Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation](https://mlsanity.com/paper/2605.13554/) (2026)

## Abstract (arXiv)

For groups of autonomous agents to achieve a particular goal, they must engage in coordination and long-horizon reasoning. Rather than relying on complex reward functions and explicit cooperation mechanisms, we ask what minimal ingredients are required for effective coordination and exploration to emerge in multi-agent settings. We investigate this question through self-supervised goal-reaching, where agents aim to maximize the likelihood of visiting a goal state rather than maximizing a reward. Despite a sparse feedback signal, we present empirical results that show self-supervised goal-reaching techniques enable agents to learn from such feedback. On MARL benchmarks, self-supervised goal-reaching outperforms alternative approaches that have access to the same sparse reward signal. Furthermore, we empirically demonstrate that multi-agent self-supervised goal-reaching approaches can be more robust than single-agent strategies. While there is no explicit exploration mechanism, this approach explores nontrivial intermediate coordination strategies in sparse settings where alternative approaches fail to achieve a single success.