# Heterogeneous Multi-agent Zero-Shot Coordination by Coevolution

arXiv: https://arxiv.org/abs/2208.04957 · HTML: https://mlsanity.com/paper/2208.04957/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 70 references that are in the map.
- First posted on arXiv 9 Aug 2022.

## Builds on (most cited first, 30 of 34)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Automatic Curriculum Learning For Deep RL: A Short Survey](https://mlsanity.com/paper/2003.04664/) (2020)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents](https://mlsanity.com/paper/1712.06560/) (2017)
- [Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions](https://mlsanity.com/paper/2003.08536/) (2020)
- [One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL](https://mlsanity.com/paper/2010.14484/) (2020)
- [Off-Belief Learning](https://mlsanity.com/paper/2103.04000/) (2021)
- [Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2112.11701/) (2021)
- [Quality-Diversity Optimization: a novel branch of stochastic optimization](https://mlsanity.com/paper/2012.04322/) (2020)
- [Illuminating Mario Scenes in the Latent Space of a Generative Adversarial Network](https://mlsanity.com/paper/2007.05674/) (2020)
- [A Survey of Ad Hoc Teamwork Research](https://mlsanity.com/paper/2202.10450/) (2022)
- [On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)](https://mlsanity.com/paper/2109.04024/) (2021)
- [Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination](https://mlsanity.com/paper/2201.12436/) (2022)
- [Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian](https://mlsanity.com/paper/2011.06505/) (2020)
- [Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning](https://mlsanity.com/paper/2108.03803/) (2021)

## Built on by (most cited first, 12 of 12)

- [Cooperative Open-ended Learning Framework for Zero-shot Coordination](https://mlsanity.com/paper/2302.04831/) (2023)
- [ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination](https://mlsanity.com/paper/2310.05208/) (2023)
- [Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers](https://mlsanity.com/paper/2305.05909/) (2023)
- [Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2306.03034/) (2023)
- [Multi-agent Continual Coordination via Progressive Task Contextualization](https://mlsanity.com/paper/2305.13937/) (2023)
- [Efficient Human-AI Coordination via Preparatory Language-based Convention](https://mlsanity.com/paper/2311.00416/) (2023)
- [Communication-Robust Multi-Agent Learning by Adaptable Auxiliary Multi-Agent Adversary Generation](https://mlsanity.com/paper/2305.05116/) (2023)
- [The Yokai Learning Environment: Tracking Beliefs Over Space and Time](https://mlsanity.com/paper/2508.12480/) (2025)
- [HOLA-Drone: Hypergraphic Open-ended Learning for Zero-Shot Multi-Drone Cooperative Pursuit](https://mlsanity.com/paper/2409.08767/) (2024)
- [Generalizable Agent Modeling for Agent Collaboration-Competition Adaptation with Multi-Retrieval and Dynamic Generation](https://mlsanity.com/paper/2506.16718/) (2025)
- [CooT: Learning to Coordinate In-Context with Coordination Transformers](https://mlsanity.com/paper/2506.23549/) (2025)
- [Zero Shot Coordination for Sparse Reward Tasks with Diverse Reward Shapings](https://mlsanity.com/paper/2604.25076/) (2026)

## Abstract (arXiv)

Generating agents that can achieve zero-shot coordination (ZSC) with unseen partners is a new challenge in cooperative multi-agent reinforcement learning (MARL). Recently, some studies have made progress in ZSC by exposing the agents to diverse partners during the training process. They usually involve self-play when training the partners, implicitly assuming that the tasks are homogeneous. However, many real-world tasks are heterogeneous, and hence previous methods may be inefficient. In this paper, we study the heterogeneous ZSC problem for the first time and propose a general method based on coevolution, which coevolves two populations of agents and partners through three sub-processes: pairing, updating and selection. Experimental results on various heterogeneous tasks highlight the necessity of considering the heterogeneous setting and demonstrate that our proposed method is a promising solution for heterogeneous ZSC tasks.