# Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition

arXiv: https://arxiv.org/abs/2105.08692 · HTML: https://mlsanity.com/paper/2105.08692/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 18 May 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [ROMA: Multi-Agent Reinforcement Learning with Emergent Roles](https://mlsanity.com/paper/2003.08039/) (2020)
- [On Improving Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1804.06309/) (2018)
- [Learning Transferable Cooperative Behavior in Multi-Agent Teams](https://mlsanity.com/paper/1906.01202/) (2019)
- [Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery](https://mlsanity.com/paper/1912.03558/) (2019)
- [From Few to More: Large-scale Dynamic Multiagent Curriculum Learning](https://mlsanity.com/paper/1909.02790/) (2019)
- [Multi-Agent Common Knowledge Reinforcement Learning](https://mlsanity.com/paper/1810.11702/) (2018)
- [M3RL: Mind-aware Multi-agent Management Reinforcement Learning](https://mlsanity.com/paper/1810.00147/) (2018)
- [A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning](https://mlsanity.com/paper/1903.06372/) (2019)
- [Distributed Policy Evaluation Under Multiple Behavior Strategies](https://mlsanity.com/paper/1312.7606/) (2013)
- [Multi-Agent Collaboration via Reward Attribution Decomposition](https://mlsanity.com/paper/2010.08531/) (2020)
- [A Structured Prediction Approach for Generalization in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1910.08809/) (2019)

## Built on by (most cited first, 30 of 30)

- [Multi-Agent Collaboration via Evolving Orchestration](https://mlsanity.com/paper/2505.19591/) (2025)
- [Learning Multi-Agent Communication from Graph Modeling Perspective](https://mlsanity.com/paper/2405.08550/) (2024)
- [RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization](https://mlsanity.com/paper/2311.01753/) (2023)
- [ALMA: Hierarchical Learning for Composite Multi-Agent Tasks](https://mlsanity.com/paper/2205.14205/) (2022)
- [N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2404.10740/) (2024)
- [Learning Multi-Agent Loco-Manipulation for Long-Horizon Quadrupedal Pushing](https://mlsanity.com/paper/2411.07104/) (2024)
- [PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.08872/) (2022)
- [AI-enhanced Collective Intelligence](https://mlsanity.com/paper/2403.10433/) (2024)
- [Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation](https://mlsanity.com/paper/2503.21200/) (2025)
- [Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.12988/) (2021)
- [Variational Offline Multi-agent Skill Discovery](https://mlsanity.com/paper/2405.16386/) (2024)
- [Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2410.08540/) (2024)
- [DM2: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching](https://mlsanity.com/paper/2206.00233/) (2022)
- [An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks](https://mlsanity.com/paper/2502.04773/) (2025)
- [Nucleolus Credit Assignment for Effective Coalitions in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2503.00372/) (2025)
- [ToolBrain: A Flexible Reinforcement Learning Framework for Agentic Tools](https://mlsanity.com/paper/2510.00023/) (2025)
- [Meta-CPR: Generalize to Unseen Large Number of Agents with Communication Pattern Recognition Module](https://mlsanity.com/paper/2112.07222/) (2021)
- [The StarCraft Multi-Agent Challenges+ : Learning of Multi-Stage Tasks and Environmental Factors without Precise Reward Functions](https://mlsanity.com/paper/2207.02007/) (2022)
- [Transferable Multi-Agent Reinforcement Learning with Dynamic Participating Agents](https://mlsanity.com/paper/2208.02424/) (2022)
- [Graded-Q Reinforcement Learning with Information-Enhanced State Encoder for Hierarchical Collaborative Multi-Vehicle Pursuit](https://mlsanity.com/paper/2210.13470/) (2022)
- [Hierarchical Strategies for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.07397/) (2022)
- [CORD: Generalizable Cooperation via Role Diversity](https://mlsanity.com/paper/2501.02221/) (2025)
- [Offsetting Unequal Competition through RL-assisted Incentive Schemes](https://mlsanity.com/paper/2201.01450/) (2022)
- [Communication Learning in Multi-Agent Systems from Graph Modeling Perspective](https://mlsanity.com/paper/2411.00382/) (2024)
- [ICCO: Learning an Instruction-conditioned Coordinator for Language-guided Task-aligned Multi-robot Control](https://mlsanity.com/paper/2503.12122/) (2025)
- [Learning Bilateral Team Formation in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2506.20039/) (2025)
- [Local-Canonicalization Equivariant Graph Neural Networks for Sample-Efficient and Generalizable Swarm Robot Control](https://mlsanity.com/paper/2509.14431/) (2025)
- [HCPO: Hierarchical Conductor-Based Policy Optimization in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2511.12123/) (2025)
- [PC3D: Zero-Shot Cooperation Across Variable Rosters via Personalized Context Distillation](https://mlsanity.com/paper/2605.10377/) (2026)
- [Turnover-Orthogonal Credit Assignment for Open-Team Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2610.02847/) (2026)

## Abstract (arXiv)

In real-world multi-agent systems, agents with different capabilities may join or leave without altering the team's overarching goals. Coordinating teams with such dynamic composition is challenging: the optimal team strategy varies with the composition. We propose COPA, a coach-player framework to tackle this problem. We assume the coach has a global view of the environment and coordinates the players, who only have partial views, by distributing individual strategies. Specifically, we 1) adopt the attention mechanism for both the coach and the players; 2) propose a variational objective to regularize learning; and 3) design an adaptive communication method to let the coach decide when to communicate with the players. We validate our methods on a resource collection task, a rescue game, and the StarCraft micromanagement tasks. We demonstrate zero-shot generalization to new team compositions. Our method achieves comparable or better performance than the setting where all players have a full view of the environment. Moreover, we see that the performance remains high even when the coach communicates as little as 13% of the time using the adaptive communication strategy.