mlsanityData to 7 Oct 2026

Quantifying the effects of environment and population diversity in multi-agent reinforcement learning

arXiv 2102.08370 ↗First posted 16 Feb 2021cs.MA cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Generalization is a major challenge for multi-agent reinforcement learning. How well does an agent perform when placed in novel environments and in interactions with new co-players? In this paper, we investigate and quantify the relationship between generalization and diversity in the multi-agent domain. Across the range of multi-agent environments considered here, procedurally generating training levels significantly improves agent performance on held-out levels. However, agent performance on the specific levels used in training sometimes declines as a result. To better understand the effects of co-player variation, our experiments introduce a new environment-agnostic measure of behavioral diversity. Results demonstrate that population size and intrinsic motivation are both effective methods of generating greater population diversity. In turn, training with a diverse set of co-players strengthens agent performance in some (but not all) cases.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 28 of its 61 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

28 of 61 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  5. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  6. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  7. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  8. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  9. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  10. 2020"Other-Play" for Zero-Shot Coordinationcited by 194
  11. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  12. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
16 more
  1. 2020Open Problems in Cooperative AIcited by 167
  2. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  3. 2018Inequity aversion improves cooperation in intertemporal social dilemmascited by 106
  4. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  5. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  6. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  7. 2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 80
  8. 2017A multi-agent reinforcement learning model of common-pool resource appropriationcited by 68
  9. 2020Real World Games Look Like Spinning Topscited by 66
  10. 2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
  11. 2020Social diversity and social preferences in mixed-motive reinforcement learningcited by 49
  12. 2021Evaluating the Robustness of Collaborative Agentscited by 25
  13. 2018Malthusian Reinforcement Learningcited by 14
  14. 2018Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learningcited by 13
  15. 2020DeepMind Lab2Dcited by 12
  16. 2020Should I tear down this wall? Optimizing social metrics by evaluating novel actionscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 7; 2025: 5; 2026: 1.

21 papers.

Sort:

By area:

  1. 2024A Survey on Context-Aware Multi-Agent Systems: Techniques, Challenges and Future Directionscited by 27
  2. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 23
  3. 2024A social path to human-like artificial intelligencecited by 20
  4. 2023Adaptive Coordination in Social Embodied Rearrangementcited by 14
  5. 2025Cross-environment Cooperation Enables Zero-shot Multi-agent Coordinationcited by 14
  6. 2023System Neural Diversity: Measuring Behavioral Heterogeneity in Multi-Agent Learningcited by 12
  7. 2022Warmth and competence in human-agent cooperationcited by 11
  8. 2024Controlling Behavioral Diversity in Multi-Agent Reinforcement Learningcited by 10
  9. 2024HyperMARL: Adaptive Hypernetworks for Multi-Agent RLcited by 10
  10. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  11. 2023RaidEnv: Exploring New Challenges in Automated Content Balancing for Boss Raid Gamescited by 7
  12. 2025Predicting Multi-Agent Specialization via Task Parallelizabilitycited by 5
9 more
  1. 2024Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gaincited by 4
  2. 2023Heterogeneous Social Value Orientation Leads to Meaningful Diversity in Sequential Social Dilemmascited by 2
  3. 2023Quantifying Agent Interaction in Multi-agent Reinforcement Learning for Cost-efficient Generalizationcited by 2
  4. 2024Symmetry-Breaking Augmentations for Ad Hoc Teamworkcited by 2
  5. 2025Implicitly Aligning Humans and Autonomous Agents through Shared Task Abstractionscited by 2
  6. 2025Unsupervised Partner Design Enables Robust Ad-hoc Teamworkcited by 2
  7. 2022RPM: Generalizable Behaviors for Multi-Agent Reinforcement Learningcited by 1
  8. 2024Dynamics of Moral Behavior in Heterogeneous Populations of Learning Agentscited by 1
  9. 2026Graph-SND: Sparse Aggregation for Behavioral Diversity in Multi-Agent Reinforcement Learningcited by 0