mlsanityData to 14 Oct 2026

Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning

arXiv 2509.25300 ↗First posted 29 Sep 2025cs.LG cs.AIcited by 29

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

While scaling laws for large language models (LLMs) during pre-training have been extensively studied, their behavior under reinforcement learning (RL) post-training remains largely unexplored. This paper presents a systematic empirical investigation of scaling behaviors in RL-based post-training, with a particular focus on mathematical reasoning. Based on a set of experiments across the full Qwen2.5 dense model series (0.5B to 72B), we characterize how model scale, data volume, and computational budget interact to shape performance. Our analysis leads to four key findings: 1. Larger models consistently exhibit superior learning efficiency on both compute and data metrics. 2. The relationship between test loss, compute, and data can be modeled by a predictive power-law which is robust across both base and instruction-tuned models. 3. Although larger models exhibit higher learning efficiency, the analytical learning efficiency term k(N) in the power-law reveals a latent saturation trend in learning efficiency as model size continues to increase. 4. In data-constrained regimes, repeated reuse of high-quality data proves highly effective, as final performance is primarily governed by the total number of optimization steps rather than the uniqueness of samples. Collectively, these results provide a principled foundation and practical guidelines for efficiently scaling the reasoning capabilities of LLMs through RL post-training.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 57 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

39 of 57 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  2. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning23 Jul
  3. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective17 Jun
  4. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  6. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2025LIMR: Less is More for RL Scaling17 Feb
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  12. 2023Let's Verify Step by Step31 May

Other references · 27

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  10. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  11. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  12. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
15 more
  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  3. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  4. 2025Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipelinecited by 82
  5. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 79
  6. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solvingcited by 62
  7. 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
  8. 2025Scaling Laws of Synthetic Data for Language Modelscited by 46
  9. 2023Scaling laws for single-agent reinforcement learningcited by 41
  10. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 39
  11. 2025Understanding Tool-Integrated Reasoningcited by 38
  12. 2024Code Comprehension then Auditing for Unsupervised LLM Evaluationcited by 32
  13. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  14. 2024gzip Predicts Data-dependent Scaling Lawscited by 16
  15. 2025Provable Benefits of In-Tool Learning for Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 26.

29 papers.

Sort:

By area:

  1. 2026Your Group-Relative Advantage Is Biasedcited by 34
  2. 2026Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillationcited by 19
  3. 2025Understanding LLM Reasoning for Abstractive Summarizationcited by 12
  4. 2026PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalizationcited by 11
  5. 2026IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RLcited by 6
  6. 2026Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Trainingcited by 5
  7. 2025M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimizationcited by 4
  8. 2025LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledgecited by 3
  9. 2026mSFT: Addressing Dataset Mixtures Overfitting Heterogeneously in Multi-task SFTcited by 3
  10. 2026One Sample to Rule Them All: Extreme Data Efficiency in Multidiscipline Reasoning with Reinforcement Learningcited by 2
  11. 2026Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedbackcited by 2
  12. 2026Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Keycited by 1
17 more
  1. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
  2. 2026RSI-Router: Evolving Subtask-Level LLM Routing and Skills for Cost-Efficient Agentscited by 1
  3. 2026Secure and Energy-Efficient Wireless Agentic AI Networkscited by 0
  4. 2026Continued AI Scaling Requires Repeated Efficiency Doublingscited by 0
  5. 2026Agentic Frameworks for Reasoning Tasks: An Empirical Studycited by 0
  6. 2026Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimescited by 0
  7. 2026Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priorscited by 0
  8. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  9. 2026Preference Tuning as Spectral Update Reorganizationcited by 0
  10. 2026Performance Foundations of Parallel & Distributed Reasoning Language Modelscited by 0
  11. 2026Scaling of Capability and Efficiency at Inference Time in Large Reasoning Modelscited by 0
  12. 2026Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectorscited by 0
  13. 2026Question-Specific Knowledge Graphs for Efficient Visual Reasoningcited by 0
  14. 2026How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Lawscited by 0
  15. 2026Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellerscited by 0
  16. 2026Efficient Resource-Shiftable RL Post-training with Generative Rewardscited by 0
  17. 2026TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  2. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  3. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling
  4. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
  5. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
  6. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning
  7. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  8. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling