mlsanityData to 7 Oct 2026

Scaling Laws for a Multi-Agent Reinforcement Learning Model

arXiv 2210.00849 ↗First posted 29 Sep 2022cs.LGcited by 37

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

The recent observation of neural power-law scaling relations has made a significant impact in the field of deep learning. A substantial amount of attention has been dedicated as a consequence to the description of scaling laws, although mostly for supervised learning and only to a reduced extent for reinforcement learning frameworks. In this paper we present an extensive study of performance scaling for a cornerstone reinforcement learning algorithm, AlphaZero. On the basis of a relationship between Elo rating, playing strength and power-law scaling, we train AlphaZero agents on the games Connect Four and Pentago and analyze their performance. We find that player strength scales as a power law in neural network parameter count when not bottlenecked by available compute, and as a power of compute when training optimally sized agents. We observe nearly identical scaling exponents for both games. Combining the two observed scaling laws we obtain a power law relating optimal size to compute similar to the ones observed for language models. We find that the predicted scaling of optimal neural network size fits our data for both games. This scaling law implies that previously published state-of-the-art game-playing models are significantly smaller than their optimal size, given the respective compute budgets. We also show that large AlphaZero models are more sample efficient, performing better than smaller models with the same amount of training data.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 43 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

27 of 43 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  9. 2021Scaling Vision Transformerscited by 993
  10. 2022A Generalist Agentcited by 802
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
15 more
  1. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  2. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  3. 2021Scaling Laws for Transfercited by 265
  4. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  5. 2022Multi-Game Decision Transformerscited by 212
  6. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  7. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 180
  8. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  9. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  10. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  11. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  12. 2021Scaling Scaling Laws with Board Gamescited by 62
  13. 2019α-Rank: Multi-Agent Evaluation by Evolutioncited by 50
  14. 2021Investment vs. reward in a competitive knapsack problemcited by 2
  15. 2014Pentago is a First Player Win: Strongly Solving a Game Using Parallel In-Core Retrograde Analysiscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 8; 2025: 14; 2026: 4.

37 papers.

Sort:

By area:

  1. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  3. 2022Broken Neural Scaling Lawscited by 102
  4. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  5. 2023Scaling laws for single-agent reinforcement learningcited by 39
  6. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  7. 2024Scaling Optimal LR Across Token Horizonscited by 35
  8. 2023Pretraining on the Test Set Is All You Needcited by 33
  9. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  10. 2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedbackcited by 26
  11. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 24
  12. 2023Uncovering Neural Scaling Laws in Molecular Representation Learningcited by 23
25 more
  1. 2024Scaling Laws for Pre-training Agents and World Modelscited by 23
  2. 2023Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Datacited by 19
  3. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  4. 2025Efficient Prediction of Pass@k Scaling in Large Language Modelscited by 14
  5. 2025Model Merging Scaling Laws in Large Language Modelscited by 10
  6. 2024Efficient Reinforcement Learning for Global Decision Making in the Presence of Local Agents at Scalecited by 8
  7. 2024Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representationscited by 8
  8. 2024AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Lawscited by 7
  9. 2026Scaling Laws of Global Weather Modelscited by 7
  10. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelinescited by 4
  11. 2024Understanding Model Selection For Learning In Strategic Environmentscited by 3
  12. 2025Meek Models Shall Inherit the Earthcited by 2
  13. 2026Survival Reinforcement Learning: Toward Scalable Self-Supervised RLcited by 2
  14. 2023To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiencycited by 1
  15. 2023Responsible Emergent Multi-Agent Behaviorcited by 1
  16. 2024Reorganizing attention-space geometry with expressive attentioncited by 1
  17. 2025Complexity Scaling Laws for Neural Models using Combinatorial Optimizationcited by 1
  18. 2025Small transformer architectures for task switchingcited by 1
  19. 2025A Data-Driven Discretized CS:GO Simulation Environment to Facilitate Strategic Multi-Agent Planning Researchcited by 1
  20. 2023Policy-Value Alignment and Robustness in Search-based Multi-Agent Learningcited by 0
  21. 2025Exploring the Efficacy of Meta-Learning: Unveiling Superior Data Diversity Utilization of MAML Over Pre-trainingcited by 0
  22. 2025Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependenciescited by 0
  23. 2025From generative AI to the brain: five takeawayscited by 0
  24. 2026Two AI Metrics Diverged: Will it Make All the Difference?cited by 0
  25. 2026Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Scaling Laws for Imitation Learning in Single-Agent Games
  2. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
  3. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
  4. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL
  5. 2026ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking
  6. 2025Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models
  7. 2026A Gold-Standard Study of What Makes a Lightweight Game-Playing Agent Strong
  8. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learning