# Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination

arXiv: https://arxiv.org/abs/2504.12714 · HTML: https://mlsanity.com/paper/2504.12714/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 60 references that are in the map.
- First posted on arXiv 17 Apr 2025.

## Builds on (most cited first, 30 of 34)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning](https://mlsanity.com/paper/2402.16801/) (2024)
- [Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2112.11701/) (2021)
- [XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX](https://mlsanity.com/paper/2312.12044/) (2023)
- [Finding Friend and Foe in Multi-Agent Games](https://mlsanity.com/paper/1906.02330/) (2019)
- [On the Importance of Environments in Human-Robot Coordination](https://mlsanity.com/paper/2106.10853/) (2021)
- [MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.03376/) (2023)
- [Theory of Minds: Understanding Behavior in Groups Through Inverse Planning](https://mlsanity.com/paper/1901.06085/) (2019)
- [Learning to Cooperate with Humans using Generative Agents](https://mlsanity.com/paper/2411.13934/) (2024)
- [Quantifying the effects of environment and population diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2102.08370/) (2021)
- [Stabilizing Unsupervised Environment Design with a Learned Adversary](https://mlsanity.com/paper/2308.10797/) (2023)
- [GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignment](https://mlsanity.com/paper/2403.11075/) (2024)
- [A Generalised Method for Empirical Game Theoretic Analysis](https://mlsanity.com/paper/1803.06376/) (2018)
- [Learning to Share and Hide Intentions using Information Regularization](https://mlsanity.com/paper/1808.02093/) (2018)
- [Empirical Game-Theoretic Analysis: A Survey](https://mlsanity.com/paper/2403.04018/) (2024)
- [Learning to Assist Humans without Inferring Rewards](https://mlsanity.com/paper/2411.02623/) (2024)
- [The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Design](https://mlsanity.com/paper/2406.17949/) (2024)
- [A Structured Prediction Approach for Generalization in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1910.08809/) (2019)
- [Generalization through Diversity: Improving Unsupervised Environment Design](https://mlsanity.com/paper/2301.08025/) (2023)

## Built on by (most cited first, 14 of 14)

- [Evaluating LLMs in Open-Source Games](https://mlsanity.com/paper/2512.00371/) (2025)
- [SciEducator: Scientific Video Understanding and Educating via Deming-Cycle Multi-Agent System](https://mlsanity.com/paper/2511.17943/) (2025)
- [Unsupervised Partner Design Enables Robust Ad-hoc Teamwork](https://mlsanity.com/paper/2508.06336/) (2025)
- [High entropy leads to symmetry-equivariant policies in Dec-POMDPs](https://mlsanity.com/paper/2511.22581/) (2025)
- [Fatigue-Aware Learning to Defer via Constrained Optimisation](https://mlsanity.com/paper/2604.00904/) (2026)
- [Unifying Agent Interaction and World Information for Multi-agent Coordination](https://mlsanity.com/paper/2509.25550/) (2025)
- [Shaping Zero-Shot Coordination via State Blocking](https://mlsanity.com/paper/2605.11688/) (2026)
- [NiceWebRL: a Python library for human subject experiments with reinforcement learning environments](https://mlsanity.com/paper/2508.15693/) (2025)
- [From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demos](https://mlsanity.com/paper/2511.02762/) (2025)
- [Solipsistic Superintelligence is Unlikely to be Cooperative](https://mlsanity.com/paper/2606.03237/) (2026)
- [CooT: Learning to Coordinate In-Context with Coordination Transformers](https://mlsanity.com/paper/2506.23549/) (2025)
- [NestRL: A Nested Training Regime for Mutual Adaptation in Human-AI Teaming](https://mlsanity.com/paper/2602.17737/) (2026)
- [Partner-Aware Hierarchical Skill Discovery for Robust Human-AI Collaboration](https://mlsanity.com/paper/2605.24352/) (2026)
- [Task diversity produces systematic transfer but inhibits continual reinforcement learning](https://mlsanity.com/paper/2606.00880/) (2026)

## Abstract (arXiv)

Zero-shot coordination (ZSC), the ability to adapt to a new partner in a cooperative task, is a critical component of human-compatible AI. While prior work has focused on training agents to cooperate on a single task, these specialized models do not generalize to new tasks, even if they are highly similar. Here, we study how reinforcement learning on a distribution of environments with a single partner enables learning general cooperative skills that support ZSC with many new partners on many new problems. We introduce two Jax-based, procedural generators that create billions of solvable coordination challenges. We develop a new paradigm called Cross-Environment Cooperation (CEC), and show that it outperforms competitive baselines quantitatively and qualitatively when collaborating with real people. Our findings suggest that learning to collaborate across many unique scenarios encourages agents to develop general norms, which prove effective for collaboration with different partners. Together, our results suggest a new route toward designing generalist cooperative agents capable of interacting with humans without requiring human data.