# Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination

arXiv: https://arxiv.org/abs/2201.12436 · HTML: https://mlsanity.com/paper/2201.12436/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 28 Jan 2022.

## Builds on (most cited first, 25 of 25)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Provably Efficient Maximum Entropy Exploration](https://mlsanity.com/paper/1812.02690/) (2018)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Off-Belief Learning](https://mlsanity.com/paper/2103.04000/) (2021)
- [Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1811.01458/) (2018)
- [Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1912.02288/) (2019)
- [Human-Level Performance in No-Press Diplomacy via Equilibrium Search](https://mlsanity.com/paper/2010.02923/) (2020)
- [Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi](https://mlsanity.com/paper/2107.07630/) (2021)
- [Improving Policies via Search in Cooperative Partially Observable Games](https://mlsanity.com/paper/1912.02318/) (2019)
- [Pommerman: A Multi-Agent Playground](https://mlsanity.com/paper/1809.07124/) (2018)
- [Too many cooks: Bayesian inference for coordinating multi-agent collaboration](https://mlsanity.com/paper/2003.11778/) (2020)
- [Quasi-Equivalence Discovery for Zero-Shot Emergent Communication](https://mlsanity.com/paper/2103.08067/) (2021)
- [Diverse Agents for Ad-Hoc Cooperation in Hanabi](https://mlsanity.com/paper/1907.03840/) (2019)
- [Evaluating and Modelling Hanabi-Playing Agents](https://mlsanity.com/paper/1704.07069/) (2017)
- [Automatic Bridge Bidding Using Deep Reinforcement Learning](https://mlsanity.com/paper/1607.03290/) (2016)
- [Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1908.01022/) (2019)

## Built on by (most cited first, 23 of 23)

- [ProAgent: Building Proactive Cooperative Agents with Large Language Models](https://mlsanity.com/paper/2308.11339/) (2023)
- [ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination](https://mlsanity.com/paper/2310.05208/) (2023)
- [Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity](https://mlsanity.com/paper/2207.14138/) (2022)
- [Heterogeneous Multi-agent Zero-Shot Coordination by Coevolution](https://mlsanity.com/paper/2208.04957/) (2022)
- [Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi](https://mlsanity.com/paper/2308.10284/) (2023)
- [A Hierarchical Approach to Population Training for Human-AI Collaboration](https://mlsanity.com/paper/2305.16708/) (2023)
- [ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork](https://mlsanity.com/paper/2505.23686/) (2025)
- [A Generalist Hanabi Agent](https://mlsanity.com/paper/2503.14555/) (2025)
- [Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions](https://mlsanity.com/paper/2411.01166/) (2024)
- [The Yokai Learning Environment: Tracking Beliefs Over Space and Time](https://mlsanity.com/paper/2508.12480/) (2025)
- [Implicitly Aligning Humans and Autonomous Agents through Shared Task Abstractions](https://mlsanity.com/paper/2505.04579/) (2025)
- [Shaping Zero-Shot Coordination via State Blocking](https://mlsanity.com/paper/2605.11688/) (2026)
- [In Pursuit of Predictive Models of Human Preferences Toward AI Teammates](https://mlsanity.com/paper/2503.15516/) (2025)
- [Efficient Generation of Diverse Cooperative Agents with World Models](https://mlsanity.com/paper/2506.07450/) (2025)
- [Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Review](https://mlsanity.com/paper/2507.10142/) (2025)
- [Zero-Shot Coordination in Ad Hoc Teams with Generalized Policy Improvement and Difference Rewards](https://mlsanity.com/paper/2510.16187/) (2025)
- [Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details](https://mlsanity.com/paper/2608.03644/) (2026)
- [The Convention Gap: Beyond Literal Content in Human-AI Cooperation](https://mlsanity.com/paper/2609.11489/) (2026)
- [AT-Drone: Benchmarking Adaptive Teaming in Multi-Drone Pursuit](https://mlsanity.com/paper/2502.09762/) (2025)
- [Controllable Complementarity: Subjective Preferences in Human-AI Collaboration](https://mlsanity.com/paper/2503.05455/) (2025)
- [CooT: Learning to Coordinate In-Context with Coordination Transformers](https://mlsanity.com/paper/2506.23549/) (2025)
- [Efficient Reinforcement Learning for Zero-Shot Coordination in Evolving Games](https://mlsanity.com/paper/2511.11083/) (2025)
- [What and Where in Identifying and Quantifying Conventions in Humans and AI Agents](https://mlsanity.com/paper/2610.20205/) (2026)

## Abstract (arXiv)

Cooperative artificial intelligence with human or superhuman proficiency in collaborative tasks stands at the frontier of machine learning research. Prior work has tended to evaluate cooperative AI performance under the restrictive paradigms of self-play (teams composed of agents trained together) and cross-play (teams of agents trained independently but using the same algorithm). Recent work has indicated that AI optimized for these narrow settings may make for undesirable collaborators in the real-world. We formalize an alternative criteria for evaluating cooperative AI, referred to as inter-algorithm cross-play, where agents are evaluated on teaming performance with all other agents within an experiment pool with no assumption of algorithmic similarities between agents. We show that existing state-of-the-art cooperative AI algorithms, such as Other-Play and Off-Belief Learning, under-perform in this paradigm. We propose the Any-Play learning augmentation -- a multi-agent extension of diversity-based intrinsic rewards for zero-shot coordination (ZSC) -- for generalizing self-play-based algorithms to the inter-algorithm cross-play setting. We apply the Any-Play learning augmentation to the Simplified Action Decoder (SAD) and demonstrate state-of-the-art performance in the collaborative card game Hanabi.