# Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning

arXiv: https://arxiv.org/abs/2006.10412 · HTML: https://mlsanity.com/paper/2006.10412/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 30 of 37)

- [A Survey of Ad Hoc Teamwork Research](https://mlsanity.com/paper/2202.10450/) (2022)
- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon Game](https://mlsanity.com/paper/2312.17515/) (2023)
- [Contrastive Identity-Aware Learning for Multi-Agent Value Decomposition](https://mlsanity.com/paper/2211.12712/) (2022)
- [Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity](https://mlsanity.com/paper/2207.14138/) (2022)
- [N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2404.10740/) (2024)
- [Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agents](https://mlsanity.com/paper/2308.09595/) (2023)
- [A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learning](https://mlsanity.com/paper/2210.05448/) (2022)
- [Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2306.03034/) (2023)
- [On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2111.05992/) (2021)
- [Fast Peer Adaptation with Context-aware Exploration](https://mlsanity.com/paper/2402.02468/) (2024)
- [Assessing the Impact of Distribution Shift on Reinforcement Learning Performance](https://mlsanity.com/paper/2402.03590/) (2024)
- [Open Ad Hoc Teamwork with Cooperative Game Theory](https://mlsanity.com/paper/2402.15259/) (2024)
- [ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork](https://mlsanity.com/paper/2505.23686/) (2025)
- [Learning Task Embeddings for Teamwork Adaptation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2207.02249/) (2022)
- [Communication-Efficient Desire Alignment for Embodied Agent-Human Adaptation](https://mlsanity.com/paper/2505.22503/) (2025)
- [Shapley Machine: A Game-Theoretic Framework for N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2506.11285/) (2025)
- [Deep Reinforcement Learning for Multi-Agent Interaction](https://mlsanity.com/paper/2208.01769/) (2022)
- [Inverse Attention Agents for Multi-Agent Systems](https://mlsanity.com/paper/2410.21794/) (2024)
- [Unsupervised Partner Design Enables Robust Ad-hoc Teamwork](https://mlsanity.com/paper/2508.06336/) (2025)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [Knowledge-based Reasoning and Learning under Partial Observability in Ad Hoc Teamwork](https://mlsanity.com/paper/2306.00790/) (2023)
- [Challenges in Credit Assignment for Multi-Agent Reinforcement Learning in Open Agent Systems](https://mlsanity.com/paper/2510.27659/) (2025)
- [Shaping Zero-Shot Coordination via State Blocking](https://mlsanity.com/paper/2605.11688/) (2026)
- [Ad Hoc Teamwork in the Presence of Adversaries](https://mlsanity.com/paper/2208.05071/) (2022)
- [Goals are Enough: Inducing AdHoc cooperation among unseen Multi-Agent systems in IMFs](https://mlsanity.com/paper/2310.17416/) (2023)
- [Zero-Shot Coordination in Ad Hoc Teams with Generalized Policy Improvement and Difference Rewards](https://mlsanity.com/paper/2510.16187/) (2025)
- [RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2206.01207/) (2022)
- [Knowledge-based and Data-driven Reasoning and Learning for Ad Hoc Teamwork](https://mlsanity.com/paper/2208.11556/) (2022)
- [Classifying Ambiguous Identities in Hidden-Role Stochastic Games with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.12896/) (2022)

## Abstract (arXiv)

Ad hoc teamwork is the challenging problem of designing an autonomous agent which can adapt quickly to collaborate with teammates without prior coordination mechanisms, including joint training. Prior work in this area has focused on closed teams in which the number of agents is fixed. In this work, we consider open teams by allowing agents with different fixed policies to enter and leave the environment without prior notification. Our solution builds on graph neural networks to learn agent models and joint-action value models under varying team compositions. We contribute a novel action-value computation that integrates the agent model and joint-action value model to produce action-value estimates. We empirically demonstrate that our approach successfully models the effects other agents have on the learner, leading to policies that robustly adapt to dynamic team compositions and significantly outperform several alternative methods.