# Cooperative Open-ended Learning Framework for Zero-shot Coordination

arXiv: https://arxiv.org/abs/2302.04831 · HTML: https://mlsanity.com/paper/2302.04831/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 9 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2112.11701/) (2021)
- [Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games](https://mlsanity.com/paper/2006.08555/) (2020)
- [Evaluating the Robustness of Collaborative Agents](https://mlsanity.com/paper/2101.05507/) (2021)
- [Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems](https://mlsanity.com/paper/2102.07659/) (2021)
- [Generalization in Cooperative Multi-Agent Systems](https://mlsanity.com/paper/2202.00104/) (2022)
- [A Generalised Method for Empirical Game Theoretic Analysis](https://mlsanity.com/paper/1803.06376/) (2018)
- [Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games](https://mlsanity.com/paper/2106.04958/) (2021)
- [Generating and Adapting to Diverse Ad-Hoc Cooperation Agents in Hanabi](https://mlsanity.com/paper/2004.13710/) (2020)
- [Heterogeneous Multi-agent Zero-Shot Coordination by Coevolution](https://mlsanity.com/paper/2208.04957/) (2022)
- [Self-Play PSRO: Toward Optimal Populations in Two-Player Zero-Sum Games](https://mlsanity.com/paper/2207.06541/) (2022)
- [Open-Ended Reinforcement Learning with Neural Reward Functions](https://mlsanity.com/paper/2202.08266/) (2022)

## Built on by (most cited first, 30 of 30)

- [A Survey on Large Language Model-Based Game Agents](https://mlsanity.com/paper/2404.02039/) (2024)
- [ProAgent: Building Proactive Cooperative Agents with Large Language Models](https://mlsanity.com/paper/2308.11339/) (2023)
- [LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Models](https://mlsanity.com/paper/2310.03903/) (2023)
- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systems](https://mlsanity.com/paper/2408.15971/) (2024)
- [ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination](https://mlsanity.com/paper/2310.05208/) (2023)
- [Learning to Cooperate with Humans using Generative Agents](https://mlsanity.com/paper/2411.13934/) (2024)
- [A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios](https://mlsanity.com/paper/2412.03920/) (2024)
- [Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Task](https://mlsanity.com/paper/2409.08811/) (2024)
- [Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2306.03034/) (2023)
- [The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Design](https://mlsanity.com/paper/2406.17949/) (2024)
- [Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2305.10865/) (2023)
- [Autonomous Goal Detection and Cessation in Reinforcement Learning: A Case Study on Source Term Estimation](https://mlsanity.com/paper/2409.09541/) (2024)
- [Shapley Machine: A Game-Theoretic Framework for N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2506.11285/) (2025)
- [Unsupervised Partner Design Enables Robust Ad-hoc Teamwork](https://mlsanity.com/paper/2508.06336/) (2025)
- [Adaptive Theory of Mind for LLM-based Multi-Agent Coordination](https://mlsanity.com/paper/2603.16264/) (2026)
- [KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordination](https://mlsanity.com/paper/2408.04336/) (2024)
- [CausalPlan: Empowering Efficient LLM Multi-Agent Collaboration Through Causality-Driven Planning](https://mlsanity.com/paper/2508.13721/) (2025)
- [Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork](https://mlsanity.com/paper/2605.24423/) (2026)
- [Learning Translations: Emergent Communication Pretraining for Cooperative Language Acquisition](https://mlsanity.com/paper/2402.16247/) (2024)
- [HOLA-Drone: Hypergraphic Open-ended Learning for Zero-Shot Multi-Drone Cooperative Pursuit](https://mlsanity.com/paper/2409.08767/) (2024)
- [Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Review](https://mlsanity.com/paper/2507.10142/) (2025)
- [SyncPlan: Long-Horizon LLM Coordination with Explicit Synchronization and Adaptive Correction](https://mlsanity.com/paper/2608.01652/) (2026)
- [JaxAHT: A JAX-Based Library for Ad Hoc Teamwork](https://mlsanity.com/paper/2609.13716/) (2026)
- [AT-Drone: Benchmarking Adaptive Teaming in Multi-Drone Pursuit](https://mlsanity.com/paper/2502.09762/) (2025)
- [CooT: Learning to Coordinate In-Context with Coordination Transformers](https://mlsanity.com/paper/2506.23549/) (2025)
- [Moving Out: Physically-grounded Human-AI Collaboration](https://mlsanity.com/paper/2507.18623/) (2025)
- [Topology-Driven Anti-Entanglement Control for Soft Robots](https://mlsanity.com/paper/2605.05236/) (2026)
- [Deep-Unfolded Coordination](https://mlsanity.com/paper/2606.19920/) (2026)
- [Multi-Robot Open Adaptive Teaming Across Unseen Environments, Partners, and Scales](https://mlsanity.com/paper/2607.04972/) (2026)

## Abstract (arXiv)

Zero-shot coordination in cooperative artificial intelligence (AI) remains a significant challenge, which means effectively coordinating with a wide range of unseen partners. Previous algorithms have attempted to address this challenge by optimizing fixed objectives within a population to improve strategy or behaviour diversity. However, these approaches can result in a loss of learning and an inability to cooperate with certain strategies within the population, known as cooperative incompatibility. To address this issue, we propose the Cooperative Open-ended LEarning (COLE) framework, which constructs open-ended objectives in cooperative games with two players from the perspective of graph theory to assess and identify the cooperative ability of each strategy. We further specify the framework and propose a practical algorithm that leverages knowledge from game theory and graph theory. Furthermore, an analysis of the learning process of the algorithm shows that it can efficiently overcome cooperative incompatibility. The experimental results in the Overcooked game environment demonstrate that our method outperforms current state-of-the-art methods when coordinating with different-level partners. Our demo is available at https://sites.google.com/view/cole-2023.