# Shapley Machine: A Game-Theoretic Framework for N-Agent Ad Hoc Teamwork

arXiv: https://arxiv.org/abs/2506.11285 · HTML: https://mlsanity.com/paper/2506.11285/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 45 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 20 of 20)

- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [Multi-Agent Risks from Advanced AI](https://mlsanity.com/paper/2502.14143/) (2025)
- [Learning Latent Representations to Influence Multi-Agent Interaction](https://mlsanity.com/paper/2011.06619/) (2020)
- [Shapley Q-value: A Local Reward Approach to Solve Global Reward Games](https://mlsanity.com/paper/1907.05707/) (2019)
- [Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks](https://mlsanity.com/paper/2110.14300/) (2021)
- [A Game-Theoretic Model and Best-Response Learning Method for Ad Hoc Coordination in Multiagent Systems](https://mlsanity.com/paper/1506.01170/) (2015)
- [Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00285/) (2021)
- [Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning](https://mlsanity.com/paper/2006.10412/) (2020)
- [Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation](https://mlsanity.com/paper/2211.02127/) (2022)
- [Cooperative Open-ended Learning Framework for Zero-shot Coordination](https://mlsanity.com/paper/2302.04831/) (2023)
- [Deep Interactive Bayesian Reinforcement Learning via Meta-Learning](https://mlsanity.com/paper/2101.03864/) (2021)
- [SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning](https://mlsanity.com/paper/2105.15013/) (2021)
- [N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2404.10740/) (2024)
- [Open Ad Hoc Teamwork with Cooperative Game Theory](https://mlsanity.com/paper/2402.15259/) (2024)
- [Multiagent Model-based Credit Assignment for Continuous Control](https://mlsanity.com/paper/2112.13937/) (2021)

## Built on by (most cited first, 4 of 4)

- [PACT: Phenotype-Aware Contrastive Team Representation for Multi-Phenotype Grouped Ad Hoc Teamwork](https://mlsanity.com/paper/2510.25340/) (2025)
- [Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents](https://mlsanity.com/paper/2511.10687/) (2025)
- [Market Regime Council for Dynamic Credit Assignment in Multi-Agent LLM Decision Systems](https://mlsanity.com/paper/2605.24490/) (2026)
- [Multi-Robot Open Adaptive Teaming Across Unseen Environments, Partners, and Scales](https://mlsanity.com/paper/2607.04972/) (2026)

## Abstract (arXiv)

Open multi-agent systems are increasingly important in modeling real-world applications, such as smart grids, swarm robotics, etc. In this paper, we aim to investigate a recently proposed problem for open multi-agent systems, referred to as n-agent ad hoc teamwork (NAHT), where only a number of agents are controlled. Existing methods tend to be based on heuristic design and consequently lack theoretical rigor and ambiguous credit assignment among agents. To address these limitations, we model and solve NAHT through the lens of cooperative game theory. More specifically, we first model an open multi-agent system, characterized by its value, as an instance situated in a space of cooperative games, generated by a set of basis games. We then extend this space, along with the state space, to accommodate dynamic scenarios, thereby characterizing NAHT. Exploiting the justifiable assumption that basis game values correspond to a sequence of n-step returns with different horizons, we represent the state values for NAHT in a form similar to λ-returns. Furthermore, we derive Shapley values to allocate state values to the controlled agents, as credits for their contributions to the ad hoc team. Different from the conventional approach to shaping Shapley values in an explicit form, we shape Shapley values by fulfilling the three axioms uniquely describing them, well defined on the extended game space describing NAHT. To estimate Shapley values in dynamic scenarios, we propose a TD(λ)-like algorithm. The resulting reinforcement learning (RL) algorithm is referred to as Shapley Machine. To our best knowledge, this is the first time that the concepts from cooperative game theory are directly related to RL concepts. In experiments, we demonstrate the effectiveness of Shapley Machine and verify reasonableness of our theory.