Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Recent advancements in reasoning-focused language models such as OpenAI's O1 and DeepSeek-R1 have shown that scaling test-time computation-through chain-of-thought reasoning and iterative exploration-can yield substantial improvements on complex tasks like mathematics and code generation. These breakthroughs have been driven by large-scale reinforcement learning (RL), particularly when combined with verifiable reward signals that provide objective and grounded supervision. In this report, we investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains. Our work identifies several key ingredients for effective training, including the use of verifiable reward tasks, enhancements to Group Relative Policy Optimization (GRPO), and practical techniques to improve training stability and generalization. We introduce controlled KL regularization, clipping ratio, and periodic reference policy resets as critical components for unlocking long-term performance gains. Our model achieves significant improvements over strong baselines, including +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks. To facilitate continued research, we release our model publicly.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 4 of its 33 references that are in the map. First posted on arXiv 16 Jul 2025.
Builds on
4 of 33 references.
Closest ancestors · 2
Same direction, newest first
- 2025Process Reinforcement through Implicit Rewards3 Feb
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
Other references · 2
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2024OpenAI o1 System Cardcited by 1,244
Built on by
Citing papers per year: 2025: 4; 2026: 1.
5 papers.
- 2025Revisiting LLM Reasoning via Information Bottleneckcited by 19
- 2025First Try Matters: Revisiting the Role of Reflection in Reasoning Modelscited by 12
- 2025Learning to Reason as Action Abstractions with Scalable Mid-Training RLcited by 3
- 2025Learning to Reason Efficiently with Discounted Reinforcement Learningcited by 3
- 2026Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Modelscited by 2
- 2026Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models11 Apr 2026
- 2025Learning to Reason Efficiently with Discounted Reinforcement Learning27 Oct 2025
- 2025First Try Matters: Revisiting the Role of Reflection in Reasoning Models9 Oct 2025
- 2025Learning to Reason as Action Abstractions with Scalable Mid-Training RL30 Sep 2025
- 2025Revisiting LLM Reasoning via Information Bottleneck24 Jul 2025
Similar, not cited
Close in wording, no citation either way
- 2025A Survey of Reinforcement Learning for Large Reasoning Models
- 2025Demystifying Long Chain-of-Thought Reasoning in LLMs
- 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
- 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
- 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
- 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
- 2025How Difficulty-Aware Staged Reinforcement Learning Enhances LLMs' Reasoning Capabilities: A Preliminary Experimental Study
- 2025Training Language Models to Reason Efficiently