# Stable Reinforcement Learning for Efficient Reasoning

arXiv: https://arxiv.org/abs/2505.18086 · HTML: https://mlsanity.com/paper/2505.18086/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025)
- [Training Language Models to Reason Efficiently](https://mlsanity.com/paper/2502.04463/) (2025)
- [Dynamic Early Exit in Reasoning Models](https://mlsanity.com/paper/2504.15895/) (2025)
- [ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning](https://mlsanity.com/paper/2504.01296/) (2025)
- [When More is Less: Understanding Chain-of-Thought Length in LLMs](https://mlsanity.com/paper/2502.07266/) (2025)
- [Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond](https://mlsanity.com/paper/2503.10460/) (2025)
- [The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks](https://mlsanity.com/paper/2502.08235/) (2025)
- [On Designing Effective RL Reward at Training Time for LLM Reasoning](https://mlsanity.com/paper/2410.15115/) (2024)
- [S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models](https://mlsanity.com/paper/2505.07686/) (2025)
- [FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models](https://mlsanity.com/paper/2503.17287/) (2025)

## Built on by (most cited first, 27 of 27)

- [Geometric-Mean Policy Optimization](https://mlsanity.com/paper/2507.20673/) (2025)
- [Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning](https://mlsanity.com/paper/2601.09708/) (2026)
- [Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic](https://mlsanity.com/paper/2603.01162/) (2026)
- [Test-time Prompt Intervention](https://mlsanity.com/paper/2508.02511/) (2025)
- [MAPO: Mixed Advantage Policy Optimization](https://mlsanity.com/paper/2509.18849/) (2025)
- [Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes It](https://mlsanity.com/paper/2602.01826/) (2026)
- [GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts](https://mlsanity.com/paper/2509.25160/) (2025)
- [Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models](https://mlsanity.com/paper/2601.11340/) (2026)
- [Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security](https://mlsanity.com/paper/2507.22037/) (2025)
- [GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning](https://mlsanity.com/paper/2508.17850/) (2025)
- [VL Norm: Rethink Loss Aggregation in RLVR](https://mlsanity.com/paper/2509.07558/) (2025)
- [Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning](https://mlsanity.com/paper/2510.04072/) (2025)
- [On the Plasticity and Stability for Post-Training Large Language Models](https://mlsanity.com/paper/2602.06453/) (2026)
- [Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learning](https://mlsanity.com/paper/2606.02132/) (2026)
- [Entropy-Guided Reasoning Compression](https://mlsanity.com/paper/2511.14258/) (2025)
- [Better, Faster: Harnessing Self-Improvement in Large Reasoning Models](https://mlsanity.com/paper/2605.24998/) (2026)
- [Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty](https://mlsanity.com/paper/2602.12113/) (2026)
- [One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometry](https://mlsanity.com/paper/2601.22521/) (2026)
- [A State-Transition Framework for Efficient LLM Reasoning](https://mlsanity.com/paper/2602.01198/) (2026)
- [Online Causal Kalman Filtering for Stable and Effective Policy Optimization](https://mlsanity.com/paper/2602.10609/) (2026)
- [Balanced Thinking: Improving Chain of Thought Training in Vision Language Models](https://mlsanity.com/paper/2603.18656/) (2026)
- [Step-GRPO: Internalizing Dynamic Early Exit for Efficient Reasoning](https://mlsanity.com/paper/2604.16890/) (2026)
- [BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning](https://mlsanity.com/paper/2605.27293/) (2026)
- [Length Penalties Make Chain-of-Thought Less Monitorable](https://mlsanity.com/paper/2607.09786/) (2026)
- [CLORE: Content-Level Optimization for Reasoning Efficiency](https://mlsanity.com/paper/2605.22211/) (2026)
- [Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning](https://mlsanity.com/paper/2608.14290/) (2026)
- [To Think or Not to Think: Allocating Reasoning Where It Helps](https://mlsanity.com/paper/2609.29664/) (2026)

## Abstract (arXiv)

The success of Deepseek-R1 has drawn the LLM community's attention to reinforcement learning (RL) methods like GRPO. However, such rule-based 0/1 outcome reward methods lack the capability to regulate the intermediate reasoning processes during chain-of-thought (CoT) generation, leading to severe overthinking phenomena. In response, recent studies have designed reward functions to reinforce models' behaviors in producing shorter yet correct completions. Nevertheless, we observe that these length-penalty reward functions exacerbate RL training instability: as the completion length decreases, model accuracy abruptly collapses, often occurring early in training. To address this issue, we propose a simple yet effective solution GRPO-λ, an efficient and stabilized variant of GRPO, which dynamically adjusts the reward strategy by monitoring the correctness ratio among completions within each query-sampled group. A low correctness ratio indicates the need to avoid length penalty that compromises CoT quality, triggering a switch to length-agnostic 0/1 rewards that prioritize reasoning capability. A high ratio maintains length penalties to boost efficiency. Experimental results show that our approach avoids training instability caused by length penalty while maintaining the optimal accuracy-efficiency trade-off. On the GSM8K, GPQA, MATH-500, AMC 2023, and AIME 2024 benchmarks, it improves average accuracy by 1.48% while reducing CoT sequence length by 47.3%.