# Zero-Shot Coordination in Ad Hoc Teams with Generalized Policy Improvement and Difference Rewards

arXiv: https://arxiv.org/abs/2510.16187 · HTML: https://mlsanity.com/paper/2510.16187/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Fast Task Inference with Variational Intrinsic Successor Features](https://mlsanity.com/paper/1906.05030/) (2019)
- [Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement](https://mlsanity.com/paper/1901.10964/) (2019)
- [Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2112.11701/) (2021)
- [A Survey of Ad Hoc Teamwork Research](https://mlsanity.com/paper/2202.10450/) (2022)
- [Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning](https://mlsanity.com/paper/2006.10412/) (2020)
- [Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased](https://mlsanity.com/paper/2302.01605/) (2023)
- [The Utility of Explainable AI in Ad Hoc Human-Machine Teaming](https://mlsanity.com/paper/2209.03943/) (2022)
- [ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination](https://mlsanity.com/paper/2310.05208/) (2023)
- [Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination](https://mlsanity.com/paper/2201.12436/) (2022)
- [N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2404.10740/) (2024)
- [Coordination with Humans via Strategy Matching](https://mlsanity.com/paper/2210.15099/) (2022)
- [Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi](https://mlsanity.com/paper/2308.10284/) (2023)
- [Difference Rewards Policy Gradients](https://mlsanity.com/paper/2012.11258/) (2020)
- [On-the-fly Strategy Adaptation for ad-hoc Agent Coordination](https://mlsanity.com/paper/2203.08015/) (2022)
- [Assisting Unknown Teammates in Unknown Tasks: Ad Hoc Teamwork under Partial Observability](https://mlsanity.com/paper/2201.03538/) (2022)
- [Controlling Type Confounding in Ad Hoc Teamwork with Instance-wise Teammate Feedback Rectification](https://mlsanity.com/paper/2306.10944/) (2023)
- [Adaptive Agent Architecture for Real-time Human-Agent Teaming](https://mlsanity.com/paper/2103.04439/) (2021)

## Built on by (most cited first, 1 of 1)

- [Is Per-Agent Policy Composition Safe? Rethinking Successor-Feature Transfer in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2608.11658/) (2026)

## Abstract (arXiv)

Real-world multi-agent systems may require ad hoc teaming, where an agent must coordinate with other previously unseen teammates to solve a task in a zero-shot manner. Prior work often either selects a pretrained policy based on an inferred model of the new teammates or pretrains a single policy that is robust to potential teammates. Instead, we propose to leverage all pretrained policies in a zero-shot transfer setting. We formalize this problem as an ad hoc multi-agent Markov decision process and present a solution that uses two key ideas, generalized policy improvement and difference rewards, for efficient and effective knowledge transfer between different teams. We empirically demonstrate that our algorithm, Generalized Policy improvement for Ad hoc Teaming (GPAT), successfully enables zero-shot transfer to new teams in three simulated environments: cooperative foraging, predator-prey, and Overcooked. We also demonstrate our algorithm in a real-world multi-robot setting.