mlsanityData to 7 Oct 2026

Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning

arXiv 2302.03429 ↗First posted 7 Feb 2023cs.AI cs.LG cs.MAcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent advances in multi-agent reinforcement learning (MARL) allow agents to coordinate their behaviors in complex environments. However, common MARL algorithms still suffer from scalability and sparse reward issues. One promising approach to resolving them is automatic curriculum learning (ACL). ACL involves a student (curriculum learner) training on tasks of increasing difficulty controlled by a teacher (curriculum generator). Despite its success, ACL's applicability is limited by (1) the lack of a general student framework for dealing with the varying number of agents across tasks and the sparse reward problem, and (2) the non-stationarity of the teacher's task due to ever-changing student strategies. As a remedy for ACL, we introduce a novel automatic curriculum learning framework, Skilled Population Curriculum (SPC), which adapts curriculum learning to multi-agent coordination. Specifically, we endow the student with population-invariant communication and a hierarchical skill set, allowing it to learn cooperation and behavior skills from distinct tasks with varying numbers of agents. In addition, we model the teacher as a contextual bandit conditioned by student policies, enabling a team of agents to change its size while still retaining previously acquired skills. We also analyze the inherent non-stationarity of this multi-agent automatic curriculum teaching problem and provide a corresponding regret bound. Empirical results show that our method improves the performance, scalability and sample efficiency in several MARL environments.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 55 of its 71 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

55 of 71 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  5. 2019Solving Rubik's Cube with a Robot Handcited by 699
  6. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  7. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  8. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  9. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  10. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  11. 2016The Option-Critic Architecturecited by 433
  12. 2016Learning Multiagent Communication with Backpropagationcited by 355
38 more
  1. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  2. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  3. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  4. 2016Variational Intrinsic Controlcited by 262
  5. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  6. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  7. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  8. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 225
  9. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  10. 2020Prioritized Level Replaycited by 190
  11. 2018Learning Attentional Communication for Multi-Agent Cooperationcited by 170
  12. 2019Active Domain Randomizationcited by 161
  13. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  14. 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
  15. 2018Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Taskscited by 114
  16. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  17. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 112
  18. 2021Replay-Guided Adversarial Environment Designcited by 108
  19. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  20. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 96
  21. 2020Game-Theoretic Multiagent Reinforcement Learningcited by 92
  22. 2019Learning to Schedule Communication in Multi-agent Reinforcement Learningcited by 89
  23. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  24. 2019Learning Efficient Multi-agent Communication: An Information Bottleneck Approachcited by 60
  25. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 58
  26. 2019Emergent Coordination Through Competitioncited by 57
  27. 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
  28. 2020Self-Paced Deep Reinforcement Learningcited by 49
  29. 2020Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learningcited by 47
  30. 2019Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discoverycited by 42
  31. 2021UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformerscited by 39
  32. 2020Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learningcited by 38
  33. 2019From Few to More: Large-scale Dynamic Multiagent Curriculum Learningcited by 36
  34. 2018Learning Curriculum Policies for Reinforcement Learningcited by 34
  35. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  36. 2021Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problemscited by 25
  37. 2018Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learningcited by 23
  38. 2021TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrationscited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024Multi-Agent Diagnostics for Robustness via Illuminated Diversitycited by 4
  2. 2025Causal-Paced Deep Reinforcement Learningcited by 1
  3. 2025Robust Agents in Open-Ended Worldscited by 1
  4. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 0