mlsanityData to 7 Oct 2026

Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems

arXiv 2111.04613 ↗First posted 8 Nov 2021cs.LG cs.MAcited by 25

Not on the map. Why?

Abstract (arXiv)

We introduce a curriculum learning algorithm, Variational Automatic Curriculum Learning (VACL), for solving challenging goal-conditioned cooperative multi-agent reinforcement learning problems. We motivate our paradigm through a variational perspective, where the learning objective can be decomposed into two terms: task learning on the current task distribution, and curriculum update to a new task distribution. Local optimization over the second term suggests that the curriculum should gradually expand the training tasks from easy to hard. Our VACL algorithm implements this variational paradigm with two practical components, task expansion and entity progression, which produces training curricula over both the task configurations as well as the number of entities in the task. Experiment results show that VACL solves a collection of sparse-reward problems with a large number of agents. Particularly, using a single desktop machine, VACL achieves 98% coverage rate with 100 agents in the simple-spread benchmark and reproduces the ramp-use behavior originally shown in OpenAI's hide-and-seek project. Our project website is at https://sites.google.com/view/vacl-neurips-2021.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 8 Nov 2021.

Builds on

30 of 36 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  5. 2019Solving Rubik's Cube with a Robot Handcited by 699
  6. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  7. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  8. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  9. 2017Teacher-Student Curriculum Learningcited by 197
  10. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
  11. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  12. 2018Learning Attentional Communication for Multi-Agent Cooperationcited by 170
18 more
  1. 2019Active Domain Randomizationcited by 161
  2. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  3. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  4. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  5. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  6. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  7. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  8. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 58
  9. 2019Exploration via Hindsight Goal Generationcited by 53
  10. 2020Self-Paced Deep Reinforcement Learningcited by 49
  11. 2020Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learningcited by 47
  12. 2019From Few to More: Large-scale Dynamic Multiagent Curriculum Learningcited by 36
  13. 2019Unsupervised Curricula for Visual Meta-Reinforcement Learningcited by 33
  14. 2019Self-Paced Contextual Reinforcement Learningcited by 26
  15. 2018BaRC: Backward Reachability Curriculum for Robotic Reinforcement Learningcited by 21
  16. 2020Adaptive Procedural Task Generation for Hard-Exploration Problemscited by 20
  17. 2020Embodied Visual Navigation with Automatic Curriculum Learning in Real Environmentscited by 20
  18. 2020Solving Hard AI Planning Instances Using Curriculum-Driven Deep Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 8; 2024: 3; 2025: 5; 2026: 3.

25 papers.

Sort:

By area:

  1. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  2. 2022Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptationcited by 28
  3. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
  4. 2023Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Explorationcited by 17
  5. 2024Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignmentcited by 9
  6. 2023On the Benefit of Optimal Transport for Curriculum Reinforcement Learningcited by 6
  7. 2022Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learningcited by 5
  8. 2025Align to Structure: Aligning Large Language Models with Structural Informationcited by 5
  9. 2023Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learningcited by 4
  10. 2023Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learningcited by 4
  11. 2022Learning Progress Driven Multi-Agent Curriculumcited by 3
  12. 2022Active Task Randomization: Learning Robust Skills via Unsupervised Generation of Diverse and Feasible Taskscited by 3
13 more
  1. 2025Heterogeneous Adversarial Play in Interactive Environmentscited by 3
  2. 2025SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networkscited by 2
  3. 2022Reinforcement Learning with Success Induced Task Prioritizationcited by 1
  4. 2023Reward-Machine-Guided, Self-Paced Reinforcement Learningcited by 1
  5. 2023Tracking Control for a Spherical Pendulum via Curriculum Reinforcement Learningcited by 1
  6. 2023Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learningcited by 1
  7. 2023A Dual Curriculum Learning Framework for Multi-UAV Pursuit-Evasion in Diverse Environmentscited by 1
  8. 2024Self-Clustering Hierarchical Multi-Agent Reinforcement Learning with Extensible Cooperation Graphcited by 1
  9. 2025Curriculum Imitation Learning of Distributed Multi-Robot Policiescited by 1
  10. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 0
  11. 2026Self Paced Gaussian Contextual Reinforcement Learningcited by 0
  12. 2026Curriculum reinforcement learning with measurable task representation learningcited by 0
  13. 2026SpikeCredit: Temporal Credit Carrier for Reinforcement Learning with Sparse Rewardscited by 0