# Towards a Better Understanding of Learning with Multiagent Teams

arXiv: https://arxiv.org/abs/2306.16205 · HTML: https://mlsanity.com/paper/2306.16205/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 49 references that are in the map.
- First posted on arXiv 28 Jun 2023.

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Emergent Tool Use From Multi-Agent Autocurricula](https://mlsanity.com/paper/1909.07528/) (2019)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [RUDDER: Return Decomposition for Delayed Rewards](https://mlsanity.com/paper/1806.07857/) (2018)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [Learning to Incentivize Other Learning Agents](https://mlsanity.com/paper/2006.06051/) (2020)
- [On the Maximum Hessian Eigenvalue and Generalization](https://mlsanity.com/paper/2206.10654/) (2022)
- [Social diversity and social preferences in mixed-motive reinforcement learning](https://mlsanity.com/paper/2002.02325/) (2020)
- [Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agents](https://mlsanity.com/paper/1903.00784/) (2019)
- [Hierarchical Reinforcement Learning with Timed Subgoals](https://mlsanity.com/paper/2112.03100/) (2021)
- [An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning](https://mlsanity.com/paper/2103.06224/) (2021)
- [D3C: Reducing the Price of Anarchy in Multi-Agent Learning](https://mlsanity.com/paper/2010.00575/) (2020)
- [Birds of a Feather Flock Together: A Close Look at Cooperation Emergence via Multi-Agent RL](https://mlsanity.com/paper/2104.11455/) (2021)
- [The Importance of Credo in Multiagent Learning](https://mlsanity.com/paper/2204.07471/) (2022)
- [Negative Update Intervals in Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1809.05096/) (2018)
- [Exploring the Benefits of Teams in Multiagent Learning](https://mlsanity.com/paper/2205.02328/) (2022)
- [Learning to Learn Group Alignment: A Self-Tuning Credo Framework with Multiagent Teams](https://mlsanity.com/paper/2304.07337/) (2023)

## Built on by (most cited first, 2 of 2)

- [Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmas](https://mlsanity.com/paper/2402.17270/) (2024)
- [Emergent Dominance Hierarchies in Reinforcement Learning Agents](https://mlsanity.com/paper/2401.12258/) (2024)

## Abstract (arXiv)

While it has long been recognized that a team of individual learning agents can be greater than the sum of its parts, recent work has shown that larger teams are not necessarily more effective than smaller ones. In this paper, we study why and under which conditions certain team structures promote effective learning for a population of individual learning agents. We show that, depending on the environment, some team structures help agents learn to specialize into specific roles, resulting in more favorable global results. However, large teams create credit assignment challenges that reduce coordination, leading to large teams performing poorly compared to smaller ones. We support our conclusions with both theoretical analysis and empirical results.