mlsanityData to 7 Oct 2026

Neural Auto-Curricula

arXiv 2106.02745 ↗First posted 4 Jun 2021cs.AI cs.MAcited by 2

Not on the map. Why?

Abstract (arXiv)

When solving two-player zero-sum games, multi-agent reinforcement learning (MARL) algorithms often create populations of agents where, at each iteration, a new agent is discovered as the best response to a mixture over the opponent population. Within such a process, the update rules of "who to compete with" (i.e., the opponent mixture) and "how to beat them" (i.e., finding best responses) are underpinned by manually developed game theoretical principles such as fictitious play and Double Oracle. In this paper, we introduce a novel framework -- Neural Auto-Curricula (NAC) -- that leverages meta-gradient descent to automate the discovery of the learning update rule without explicit human design. Specifically, we parameterise the opponent selection module by neural networks and the best-response module by optimisation subroutines, and update their parameters solely via interaction with the game engine, where both players aim to minimise their exploitability. Surprisingly, even without human design, the discovered MARL algorithms achieve competitive or even better performance with the state-of-the-art population-based game solvers (e.g., PSRO) on Games of Skill, differentiable Lotto, non-transitive Mixture Games, Iterated Matching Pennies, and Kuhn Poker. Additionally, we show that NAC is able to generalise from small games to large games, for example training on Kuhn Poker and outperforming PSRO on Leduc Poker. Our work inspires a promising future direction to discover general MARL algorithms solely from data.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 49 of its 71 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

49 of 71 references.

References · 49

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  6. 2017Deep Setscited by 1,394
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  9. 2019Meta-Learning with Implicit Gradientscited by 523
  10. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  11. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  12. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
37 more
  1. 2016Learning to reinforcement learncited by 369
  2. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  3. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  4. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  5. 2017Learning with Opponent-Learning Awarenesscited by 218
  6. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 180
  7. 2018Meta-Gradient Reinforcement Learningcited by 156
  8. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  9. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 106
  10. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 100
  11. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  12. 2020Game-Theoretic Multiagent Reinforcement Learningcited by 92
  13. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  14. 2018Evolved Policy Gradientscited by 86
  15. 2020Discovering Reinforcement Learning Algorithmscited by 75
  16. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  17. 2020Real World Games Look Like Spinning Topscited by 66
  18. 2019Meta-Learning via Learned Losscited by 60
  19. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 58
  20. 2019ES-MAML: Simple Hessian-Free Meta Learningcited by 57
  21. 2019A Generalized Training Approach for Multiagent Learningcited by 53
  22. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  23. 2018Structured Evolution with Compact Architectures for Scalable Policy Optimizationcited by 48
  24. 2020Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Gamescited by 47
  25. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
  26. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  27. 2019A Deterministic Linear Program Solver in Current Matrix Multiplication Timecited by 42
  28. 2020A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learningcited by 40
  29. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  30. 2019What Can Learned Intrinsic Rewards Capture?cited by 36
  31. 2020Multi-Agent Determinantal Q-Learningcited by 36
  32. 2021XDO: A Double Oracle Algorithm for Extensive-Form Gamescited by 34
  33. 2021Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systemscited by 24
  34. 2021Online Double Oraclecited by 18
  35. 2020Online Meta-Critic Learning for Off-Policy Actor-Critic Methodscited by 17
  36. 2019αα-Rank: Practically Scaling α-Rank through Stochastic Optimisationcited by 10
  37. 2021Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  2. 2026Self Paced Gaussian Contextual Reinforcement Learningcited by 0