mlsanityData to 14 Oct 2026

Scaling laws for single-agent reinforcement learning

arXiv 2301.13442 ↗First posted 31 Jan 2023cs.LG cs.AI stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that, in generative modeling, cross-entropy loss improves smoothly with model size and training compute, following a power law plus constant scaling law. One challenge in extending these results to reinforcement learning is that the main performance objective of interest, mean episode return, need not vary smoothly. To overcome this, we introduce *intrinsic performance*, a monotonic function of the return defined as the minimum compute required to achieve the given return across a family of models of different sizes. We find that, across a range of environments, intrinsic performance scales as a power law in model size and environment interactions. Consequently, as in generative modeling, the optimal model size scales as a power law in the training compute budget. Furthermore, we study how this relationship varies with the environment and with other properties of the training setup. In particular, using a toy MNIST-based environment, we show that varying the "horizon length" of the task mostly changes the coefficient but not the exponent of this relationship.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

20 of 27 references.

References · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  10. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  11. 2021Explaining Neural Scaling Lawscited by 363
  12. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
8 more
  1. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  2. 2020Phasic Policy Gradientcited by 123
  3. 2021Scaling Laws for Neural Machine Translationcited by 111
  4. 2022Broken Neural Scaling Lawscited by 109
  5. 2021Scaling Scaling Laws with Board Gamescited by 63
  6. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 38
  7. 2021Batch size-invariance for policy optimizationcited by 35
  8. 2021Scaling Laws for Acoustic Modelscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 7; 2025: 17; 2026: 9.

41 papers.

Sort:

By area:

  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  2. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 145
  3. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  4. 2023Synthetic Experience Replaycited by 101
  5. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  6. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  7. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  8. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  9. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoningcited by 29
  10. 2023AI capabilities can be significantly improved without expensive retrainingcited by 28
  11. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 26
  12. 2024Scaling Laws for Pre-training Agents and World Modelscited by 23
29 more
  1. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
  2. 2025Efficient Prediction of Pass@k Scaling in Large Language Modelscited by 16
  3. 2025Value-Based Deep RL Scales Predictablycited by 15
  4. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 15
  5. 2023Scaling Laws Do Not Scalecited by 13
  6. 2025Model Merging Scaling Laws in Large Language Modelscited by 12
  7. 2023Scaling Laws for Imitation Learning in Single-Agent Gamescited by 11
  8. 2024AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Lawscited by 7
  9. 2023Surgical Gym: A high-performance GPU-based platform for reinforcement learning with surgical robotscited by 6
  10. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 6
  11. 2026IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RLcited by 6
  12. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelinescited by 5
  13. 2025Discovering highly efficient low-weight quantum error-correcting codes with reinforcement learningcited by 5
  14. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scalingcited by 5
  15. 2024Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelismcited by 4
  16. 2025Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Explorationcited by 3
  17. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  18. 2023Scaling Is All You Need: Autonomous Driving with JAX-Accelerated Reinforcement Learningcited by 2
  19. 2026Unified Neural Scaling Lawscited by 2
  20. 2026Small-Scale Experiments: Are We There Yet?cited by 2
  21. 2026When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysiscited by 2
  22. 2025Complexity Scaling Laws for Neural Models using Combinatorial Optimizationcited by 1
  23. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1
  24. 2023Neural scaling laws for phenotypic drug discoverycited by 0
  25. 2024Scaling Particle Collision Data Analysiscited by 0
  26. 2026On Training in Imaginationcited by 0
  27. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  28. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learningcited by 0
  29. 2026Single or Multiple Policies for Phase-Structured Reinforcement Learning?cited by 0