mlsanityData to 7 Oct 2026

Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems

arXiv 2102.07659 ↗First posted 15 Feb 2021cs.AI cs.MAcited by 24

Not on the map. Why?

Abstract (arXiv)

Multiagent reinforcement learning (MARL) has achieved a remarkable amount of success in solving various types of video games. A cornerstone of this success is the auto-curriculum framework, which shapes the learning process by continually creating new challenging tasks for agents to adapt to, thereby facilitating the acquisition of new skills. In order to extend MARL methods to real-world domains outside of video games, we envision in this blue sky paper that maintaining a diversity-aware auto-curriculum is critical for successful MARL applications. Specifically, we argue that behavioural diversity is a pivotal, yet under-explored, component for real-world multiagent learning systems, and that significant work remains in understanding how to design a diversity-aware auto-curriculum. We list four open challenges for auto-curriculum techniques, which we believe deserve more attention from this community. Towards validating our vision, we recommend modelling realistic interactive behaviours in autonomous driving as an important test bed, and recommend the SMARTS/ULTRA benchmark.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 49 of its 99 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

49 of 99 references.

References · 49

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2018Learning Dexterous In-Hand Manipulationcited by 643
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  10. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  11. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  12. 2015Illuminating search spaces by mapping elitescited by 406
37 more
  1. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  2. 2019Challenges of Real-World Reinforcement Learningcited by 282
  3. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  4. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  5. 2016Variational Intrinsic Controlcited by 262
  6. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  7. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  8. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  9. 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 206
  10. 2014Robots that can adapt like animalscited by 203
  11. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
  12. 2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 173
  13. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
  14. 2019Self-Driving Cars: A Surveycited by 147
  15. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  16. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  17. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  18. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  19. 2018The Mechanics of n-Player Differentiable Gamescited by 118
  20. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  21. 2018Inequity aversion improves cooperation in intertemporal social dilemmascited by 106
  22. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  23. 2017Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Gamescited by 93
  24. 2020Game-Theoretic Multiagent Reinforcement Learningcited by 92
  25. 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 89
  26. 2017Robust Imitation of Diverse Behaviorscited by 85
  27. 2020Towards Playing Full MOBA Games with Deep Reinforcement Learningcited by 78
  28. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75
  29. 2017A multi-agent reinforcement learning model of common-pool resource appropriationcited by 68
  30. 2020Real World Games Look Like Spinning Topscited by 66
  31. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 58
  32. 2019Emergent Coordination Through Competitioncited by 57
  33. 2019A Generalized Training Approach for Multiagent Learningcited by 53
  34. 2018Re-evaluating Evaluationcited by 47
  35. 2020Multi-Agent Determinantal Q-Learningcited by 36
  36. 2020TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Gamecited by 21
  37. 2017Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learningcited by 18

Built on by

Cited by year

Citing papers per year: 2021: 10; 2022: 3; 2023: 5; 2024: 3; 2025: 3.

24 papers.

Sort:

By area:

  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  2. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  3. 2021Multi-Agent Constrained Policy Optimisationcited by 32
  4. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
  5. 2023Cooperative Open-ended Learning Framework for Zero-shot Coordinationcited by 30
  6. 2023MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learningcited by 30
  7. 2021Online Double Oraclecited by 18
  8. 2021Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Gamescited by 14
  9. 2021A Game-Theoretic Approach for Improving Generalization Ability of TSP Solverscited by 14
  10. 2022RAPid-Learn: A Framework for Learning to Recover for Handling Novelties in Open-World Environmentscited by 13
  11. 2023Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordinationcited by 12
  12. 2021Measuring the Non-Transitivity in Chesscited by 10
12 more
  1. 2024Policy Space Response Oracles: A Surveycited by 10
  2. 2025Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Surveycited by 10
  3. 2025Strengthening Proportionality in Temporal Votingcited by 9
  4. 2022Efficient Policy Space Response Oraclescited by 8
  5. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
  6. 2021Online Markov Decision Processes with Non-oblivious Strategic Adversarycited by 3
  7. 2021Neural Auto-Curriculacited by 2
  8. 2023JiangJun: Mastering Xiangqi by Tackling Non-Transitivity in Two-Player Zero-Sum Gamescited by 2
  9. 2024Self-Clustering Hierarchical Multi-Agent Reinforcement Learning with Extensible Cooperation Graphcited by 1
  10. 2025Training RL Agents for Multi-Objective Network Defense Taskscited by 1
  11. 2021Individual and Collective Autonomous Developmentcited by 0
  12. 2024Phasic Diversity Optimization for Population-Based Reinforcement Learningcited by 0