# Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning

arXiv: https://arxiv.org/abs/2506.05256 · HTML: https://mlsanity.com/paper/2506.05256/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 17 of 17)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://mlsanity.com/paper/2410.07985/) (2024)
- [Training Language Models to Reason Efficiently](https://mlsanity.com/paper/2502.04463/) (2025)
- [Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning](https://mlsanity.com/paper/2405.00451/) (2024)
- [ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning](https://mlsanity.com/paper/2504.01296/) (2025)
- [Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems](https://mlsanity.com/paper/2412.09413/) (2024)
- [Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs](https://mlsanity.com/paper/2501.18585/) (2025)
- [Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought](https://mlsanity.com/paper/2501.04682/) (2025)
- [Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum](https://mlsanity.com/paper/2405.13226/) (2024)

## Built on by (most cited first, 30 of 52)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning](https://mlsanity.com/paper/2508.09726/) (2025)
- [Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Models](https://mlsanity.com/paper/2508.02120/) (2025)
- [Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning](https://mlsanity.com/paper/2601.09708/) (2026)
- [Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2507.09662/) (2025)
- [ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping](https://mlsanity.com/paper/2510.08457/) (2025)
- [DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization](https://mlsanity.com/paper/2510.04474/) (2025)
- [On-policy Distillation with Verifiable Reward](https://mlsanity.com/paper/2608.24696/) (2026)
- [Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model](https://mlsanity.com/paper/2512.21540/) (2025)
- [SABER: Switchable and Balanced Training for Efficient LLM Reasoning](https://mlsanity.com/paper/2508.10026/) (2025)
- [ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning](https://mlsanity.com/paper/2602.09953/) (2026)
- [Schoenfeld's Anatomy of Mathematical Reasoning by Language Models](https://mlsanity.com/paper/2512.19995/) (2025)
- [Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models](https://mlsanity.com/paper/2601.11340/) (2026)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [e1: Learning Adaptive Control of Reasoning Effort](https://mlsanity.com/paper/2510.27042/) (2025)
- [From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning](https://mlsanity.com/paper/2606.18089/) (2026)
- [Steering Large Reasoning Models towards Concise Reasoning via Flow Matching](https://mlsanity.com/paper/2602.05539/) (2026)
- [On the Optimal Reasoning Length for RL-Trained Language Models](https://mlsanity.com/paper/2602.09591/) (2026)
- [ZAYA1-8B Technical Report](https://mlsanity.com/paper/2605.05365/) (2026)
- [Libra: Efficient Resource Management for Agentic RL Post-Training](https://mlsanity.com/paper/2606.03077/) (2026)
- [Intern-S2-Preview: Scientific Agentic Foundation Model](https://mlsanity.com/paper/2608.13505/) (2026)
- [Verifying Meta-Awareness via Predictive Rewards in Reasoning Models](https://mlsanity.com/paper/2510.03259/) (2025)
- [Concise Reasoning in the Lens of Lagrangian Optimization](https://mlsanity.com/paper/2510.10168/) (2025)
- [Learning to Reason Efficiently with Discounted Reinforcement Learning](https://mlsanity.com/paper/2510.23486/) (2025)
- [ESTAR: Early-Stopping Token-Aware Reasoning For Efficient Inference](https://mlsanity.com/paper/2602.10004/) (2026)
- [Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning](https://mlsanity.com/paper/2603.00296/) (2026)
- [CODA: Difficulty-Aware Compute Allocation for Adaptive Reasoning](https://mlsanity.com/paper/2603.08659/) (2026)
- [DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2605.08441/) (2026)
- [Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewards](https://mlsanity.com/paper/2606.22716/) (2026)
- [Efficient Reasoning via Reward Model](https://mlsanity.com/paper/2511.09158/) (2025)

## Abstract (arXiv)

Large reasoning models (LRMs) achieve higher performance on challenging reasoning tasks by generating more tokens at inference time, but this verbosity often wastes computation on easy problems. Existing solutions, including supervised finetuning on shorter traces, user-controlled budgets, or RL with uniform penalties, either require data curation, manual configuration, or treat all problems alike regardless of difficulty. We introduce Adaptive Length Penalty (ALP), a reinforcement learning objective tailoring generation length to per-prompt solve rate. During training, ALP monitors each prompt's online solve rate through multiple rollouts and adds a differentiable penalty whose magnitude scales inversely with that rate, so confident (easy) prompts incur a high cost for extra tokens while hard prompts remain unhindered. Posttraining DeepScaleR-1.5B with ALP cuts average token usage by 50% without significantly dropping performance. Relative to fixed-budget and uniform penalty baselines, ALP redistributes its reduced budget more intelligently by cutting compute on easy prompts and reallocating saved tokens to difficult ones, delivering higher accuracy on the hardest problems with higher cost.