mlsanityData to 7 Oct 2026

Generalization in Cooperative Multi-Agent Systems

arXiv 2202.00104 ↗First posted 31 Jan 2022cs.LG cs.AI cs.MAcited by 18

Not on the map. Why?

Abstract (arXiv)

Collective intelligence is a fundamental trait shared by several species of living organisms. It has allowed them to thrive in the diverse environmental conditions that exist on our planet. From simple organisations in an ant colony to complex systems in human groups, collective intelligence is vital for solving complex survival tasks. As is commonly observed, such natural systems are flexible to changes in their structure. Specifically, they exhibit a high degree of generalization when the abilities or the total number of agents changes within a system. We term this phenomenon as Combinatorial Generalization (CG). CG is a highly desirable trait for autonomous systems as it can increase their utility and deployability across a wide range of applications. While recent works addressing specific aspects of CG have shown impressive results on complex domains, they provide no performance guarantees when generalizing towards novel situations. In this work, we shed light on the theoretical underpinnings of CG for cooperative multi-agent systems (MAS). Specifically, we study generalization bounds under a linear dependence of the underlying dynamics on the agent capabilities, which can be seen as a generalization of Successor Features to MAS. We then extend the results first for Lipschitz and then arbitrary dependence of rewards on team capabilities. Finally, empirical analysis on various domains using the framework of multi-agent reinforcement learning highlights important desiderata for multi-agent algorithms towards ensuring CG.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 31 Jan 2022.

Builds on

33 of 59 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  5. 2019The StarCraft Multi-Agent Challengecited by 589
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  8. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  9. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  10. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  11. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  12. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
21 more
  1. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  2. 2015Contextual Markov Decision Processescited by 213
  3. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  4. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  5. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  6. 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
  7. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  8. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 97
  9. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  10. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  11. 2020Real World Games Look Like Spinning Topscited by 66
  12. 2020Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learningcited by 38
  13. 2019Multi-Agent Actor-Critic with Hierarchical Graph Attention Networkcited by 33
  14. 2020UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learningcited by 32
  15. 2021Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learningcited by 25
  16. 2019Reasoning about Hypothetical Agent Behaviours and their Parameterscited by 24
  17. 2015Belief and Truth in Hypothesised Behaviourscited by 19
  18. 2021When Is Generalizable Reinforcement Learning Tractable?cited by 19
  19. 2017Symmetry Learning for Function Approximation in Reinforcement Learningcited by 14
  20. 2020Learning Robust State Abstractions for Hidden-Parameter Block MDPscited by 4
  21. 2021Evolutionary Dynamics and Phi-Regret Minimization in Gamescited by 2

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 6; 2024: 2; 2025: 4.

18 papers.

Sort:

By area:

  1. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  2. 2023Cooperative Open-ended Learning Framework for Zero-shot Coordinationcited by 30
  3. 2023MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learningcited by 30
  4. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
  5. 2023Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordinationcited by 12
  6. 2025Generalized Mission Planning for Heterogeneous Multi-Robot Teams via LLM-constructed Hierarchical Treescited by 9
  7. 2022Interaction Pattern Disentangling for Multi-Agent Reinforcement Learningcited by 8
  8. 2024Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planningcited by 7
  9. 2022Effects of Spectral Normalization in Multi-agent Reinforcement Learningcited by 6
  10. 2022Learning Task Embeddings for Teamwork Adaptation in Multi-Agent Reinforcement Learningcited by 5
  11. 2023Mix-ME: Quality-Diversity for Multi-Agent Learningcited by 4
  12. 2022RPM: Generalizable Behaviors for Multi-Agent Reinforcement Learningcited by 1
6 more
  1. 2023marl-jax: Multi-Agent Reinforcement Leaning Frameworkcited by 1
  2. 2024HOLA-Drone: Hypergraphic Open-ended Learning for Zero-Shot Multi-Drone Cooperative Pursuitcited by 1
  3. 2025CORD: Generalizable Cooperation via Role Diversitycited by 1
  4. 2025Robust Agents in Open-Ended Worldscited by 1
  5. 2022RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learningcited by 0
  6. 2023Generalization Across Observation Shifts in Reinforcement Learningcited by 0