mlsanityData to 7 Oct 2026

The Art of Scaling Reinforcement Learning Compute for LLMs

arXiv 2510.13786 ↗First posted 15 Oct 2025cs.LG cs.AIcited by 91

Not on the map. Why?

Cited by 43 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pre-training. Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute. We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyzing and predicting RL scaling in LLMs. We fit sigmoidal compute-performance curves for RL training and ablate a wide range of common design choices to analyze their effects on asymptotic performance and compute efficiency. We observe: (1) Not all recipes yield similar asymptotic performance, (2) Details such as loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote, and (3) Stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs. Combining these insights, we propose a best-practice recipe, ScaleRL, and demonstrate its effectiveness by successfully scaling and predicting validation performance on a single RL run scaled up to 100,000 GPU-hours. Our work provides both a scientific framework for analyzing scaling in RL and a practical recipe that brings RL training closer to the predictability long achieved in pre-training.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

37 of 49 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
25 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  5. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  6. 2023Scaling Data-Constrained Language Modelscited by 391
  7. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 281
  8. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  9. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  10. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  11. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 157
  12. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  13. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  14. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 95
  15. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 87
  16. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  17. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  18. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  19. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 53
  20. 2024Quantifying Variance in Evaluation Benchmarkscited by 52
  21. 2024How predictable is language model benchmark performance?cited by 36
  22. 2025Scalable Chain of Thoughts via Elastic Reasoningcited by 32
  23. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 27
  24. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 24
  25. 2025How to Train Your LLM Web Agent: A Statistical Diagnosiscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 83.

91 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 285
  2. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
  3. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 43
  4. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 39
  5. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
  6. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 32
  7. 2026Maximum Likelihood Reinforcement Learningcited by 29
  8. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  9. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Designcited by 17
  10. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domainscited by 17
  11. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  12. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 15
38 more
  1. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  2. 2026ARLArena: A Unified Framework for Stable Agentic Reinforcement Learningcited by 11
  3. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradientscited by 10
  4. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rarecited by 10
  5. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  6. 2026Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoningcited by 9
  7. 2026Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Textcited by 9
  8. 2026Scaling Self-Play with Self-Guidancecited by 9
  9. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal promptscited by 7
  10. 2026Agentic AI Scientists Are Not Built For Autonomous Scientific Discoverycited by 7
  11. 2026Learning, Fast and Slow: Towards LLMs That Adapt Continuallycited by 6
  12. 2026Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RLcited by 5
  13. 2026Evolutionary System Prompt Learning for Reinforcement Learning in LLMscited by 5
  14. 2026IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RLcited by 5
  15. 2026On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Lengthcited by 5
  16. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 4
  17. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 4
  18. 2026Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewardscited by 4
  19. 2026ZAYA1-8B Technical Reportcited by 4
  20. 2026Mach-Mind-4-Flash Technical Reportcited by 4
  21. 2025INTELLECT-3: Technical Reportcited by 3
  22. 2025Scaling Reinforcement Learning for Content Moderation with Large Language Modelscited by 3
  23. 2026SWE-Prot'eg'e: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agentscited by 3
  24. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  25. 2026Weblica: Scalable and Reproducible Training Environments for Visual Web Agentscited by 3
  26. 2026AsyncOPD: How Stale Can On-Policy Distillation Be?cited by 3
  27. 2025Arbitrage: Efficient Reasoning via Advantage-Aware Speculationcited by 2
  28. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoningcited by 2
  29. 2026K-EXAONE Technical Reportcited by 2
  30. 2026Scaling Medical Reasoning Verification via Tool-Integrated Reinforcement Learningcited by 2
  31. 2026Uncovering Cross-Objective Interference in Multi-Objective Alignmentcited by 2
  32. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  33. 2026PACE: Prefix-Protected and Difficulty-Aware Compression for Efficient Reasoningcited by 2
  34. 2026Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoningcited by 2
  35. 2026CUBE: A Standard for Unifying Agent Benchmarkscited by 2
  36. 2026Learning to Reason with Curriculum I: Provable Benefits of Autocurriculumcited by 2
  37. 2026Exploring Pass-Rate Reward in Reinforcement Learning for Code Generationcited by 2
  38. 2026Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPOcited by 2

50 most cited of 91 All 91 (JSON)