# Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training

arXiv: https://arxiv.org/abs/2507.12507 · HTML: https://mlsanity.com/paper/2507.12507/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 4 of its 33 references that are in the map.
- First posted on arXiv 16 Jul 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 4 of 4)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)

## Built on by (most cited first, 5 of 5)

- [Revisiting LLM Reasoning via Information Bottleneck](https://mlsanity.com/paper/2507.18391/) (2025)
- [First Try Matters: Revisiting the Role of Reflection in Reasoning Models](https://mlsanity.com/paper/2510.08308/) (2025)
- [Learning to Reason as Action Abstractions with Scalable Mid-Training RL](https://mlsanity.com/paper/2509.25810/) (2025)
- [Learning to Reason Efficiently with Discounted Reinforcement Learning](https://mlsanity.com/paper/2510.23486/) (2025)
- [Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models](https://mlsanity.com/paper/2604.10219/) (2026)

## Abstract (arXiv)

Recent advancements in reasoning-focused language models such as OpenAI's O1 and DeepSeek-R1 have shown that scaling test-time computation-through chain-of-thought reasoning and iterative exploration-can yield substantial improvements on complex tasks like mathematics and code generation. These breakthroughs have been driven by large-scale reinforcement learning (RL), particularly when combined with verifiable reward signals that provide objective and grounded supervision. In this report, we investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains. Our work identifies several key ingredients for effective training, including the use of verifiable reward tasks, enhancements to Group Relative Policy Optimization (GRPO), and practical techniques to improve training stability and generalization. We introduce controlled KL regularization, clipping ratio, and periodic reference policy resets as critical components for unlocking long-term performance gains. Our model achieves significant improvements over strong baselines, including +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks. To facilitate continued research, we release our model publicly.