# Shorten After You're Right: Lazy Length Penalties for Reasoning RL

arXiv: https://arxiv.org/abs/2505.12284 · HTML: https://mlsanity.com/paper/2505.12284/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 18 May 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 27 of 27)

- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [Chain of Draft: Thinking Faster by Writing Less](https://mlsanity.com/paper/2502.18600/) (2025)
- [TokenSkip: Controllable Chain-of-Thought Compression in LLMs](https://mlsanity.com/paper/2502.12067/) (2025)
- [Training Language Models to Reason Efficiently](https://mlsanity.com/paper/2502.04463/) (2025)
- [Dynamic Early Exit in Reasoning Models](https://mlsanity.com/paper/2504.15895/) (2025)
- [Reasoning Models Can Be Effective Without Thinking](https://mlsanity.com/paper/2504.09858/) (2025)
- [CoT-Valve: Length-Compressible Chain-of-Thought Tuning](https://mlsanity.com/paper/2502.09601/) (2025)
- [Distilling System 2 into System 1](https://mlsanity.com/paper/2407.06023/) (2024)
- [DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models](https://mlsanity.com/paper/2503.04472/) (2025)
- [C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness](https://mlsanity.com/paper/2412.11664/) (2024)
- [On Memorization of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2410.23123/) (2024)
- [Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching](https://mlsanity.com/paper/2503.05179/) (2025)
- [Learning to Route LLMs with Confidence Tokens](https://mlsanity.com/paper/2410.13284/) (2024)
- [The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models](https://mlsanity.com/paper/2401.05618/) (2024)
- [Can Language Models Learn to Skip Steps?](https://mlsanity.com/paper/2411.01855/) (2024)
- [Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging](https://mlsanity.com/paper/2503.20641/) (2025)
- [THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Models](https://mlsanity.com/paper/2504.13367/) (2025)
- [Hawkeye:Efficient Reasoning with Model Collaboration](https://mlsanity.com/paper/2504.00424/) (2025)

## Built on by (most cited first, 25 of 25)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Models](https://mlsanity.com/paper/2508.02120/) (2025)
- [Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning](https://mlsanity.com/paper/2601.09708/) (2026)
- [Interleaved Reasoning for Large Language Models via Reinforcement Learning](https://mlsanity.com/paper/2505.19640/) (2025)
- [Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2507.09662/) (2025)
- [Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexity](https://mlsanity.com/paper/2509.07339/) (2025)
- [LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning](https://mlsanity.com/paper/2510.01459/) (2025)
- [Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes It](https://mlsanity.com/paper/2602.01826/) (2026)
- [C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning](https://mlsanity.com/paper/2509.23129/) (2025)
- [Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models](https://mlsanity.com/paper/2601.11340/) (2026)
- [DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains](https://mlsanity.com/paper/2510.27419/) (2025)
- [When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning](https://mlsanity.com/paper/2605.03314/) (2026)
- [VL Norm: Rethink Loss Aggregation in RLVR](https://mlsanity.com/paper/2509.07558/) (2025)
- [LACONIC: Length-Aware Constrained Reinforcement Learning for LLM](https://mlsanity.com/paper/2602.14468/) (2026)
- [ZAYA1-8B Technical Report](https://mlsanity.com/paper/2605.05365/) (2026)
- [DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2605.08441/) (2026)
- [Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure](https://mlsanity.com/paper/2605.27996/) (2026)
- [Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewards](https://mlsanity.com/paper/2606.22716/) (2026)
- [PACE: Prefix-Protected and Difficulty-Aware Compression for Efficient Reasoning](https://mlsanity.com/paper/2602.11639/) (2026)
- [LiveThinking: Enabling Real-Time Efficient Reasoning for AI-Powered Livestreaming via Reinforcement Learning](https://mlsanity.com/paper/2510.07685/) (2025)
- [Segment-Level Attribution for Selective Learning of Long Reasoning Traces](https://mlsanity.com/paper/2602.00425/) (2026)
- [CLORE: Content-Level Optimization for Reasoning Efficiency](https://mlsanity.com/paper/2605.22211/) (2026)
- [Plan Right, Then Plan Tight: Symbolic RL for Efficient Embodied Reasoning](https://mlsanity.com/paper/2606.31260/) (2026)
- [Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learning](https://mlsanity.com/paper/2609.34718/) (2026)
- [Mitigating the Length-Scaling Tax with Online Distillation](https://mlsanity.com/paper/2609.38854/) (2026)

## Abstract (arXiv)

Large reasoning models, such as OpenAI o1 or DeepSeek R1, have demonstrated remarkable performance on reasoning tasks but often incur a long reasoning path with significant memory and time costs. Existing methods primarily aim to shorten reasoning paths by introducing additional training data and stages. In this paper, we propose three critical reward designs integrated directly into the reinforcement learning process of large reasoning models, which reduce the response length without extra training stages. Experiments on four settings show that our method significantly decreases response length while maintaining or even improving performance. Specifically, in a logic reasoning setting, we achieve a 40% reduction in response length averaged by steps alongside a 14% gain in performance. For math problems, we reduce response length averaged by steps by 33% while preserving performance.