mlsanityData to 14 Oct 2026

Shorten After You're Right: Lazy Length Penalties for Reasoning RL

arXiv 2505.12284 ↗First posted 18 May 2025cs.AI cs.CLcited by 25

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large reasoning models, such as OpenAI o1 or DeepSeek R1, have demonstrated remarkable performance on reasoning tasks but often incur a long reasoning path with significant memory and time costs. Existing methods primarily aim to shorten reasoning paths by introducing additional training data and stages. In this paper, we propose three critical reward designs integrated directly into the reinforcement learning process of large reasoning models, which reduce the response length without extra training stages. Experiments on four settings show that our method significantly decreases response length while maintaining or even improving performance. Specifically, in a logic reasoning setting, we achieve a 40% reduction in response length averaged by steps alongside a 14% gain in performance. For math problems, we reduce response length averaged by steps by 33% while preserving performance.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 35 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

27 of 35 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Dynamic Early Exit in Reasoning Models22 Apr
  2. 2025THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Models17 Apr
  3. 2025Reasoning Models Can Be Effective Without Thinking14 Apr
  4. 2025Hawkeye:Efficient Reasoning with Model Collaboration1 Apr
  5. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging26 Mar
  6. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models20 Mar
  7. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning6 Mar
  8. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models6 Mar
  9. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMs17 Feb
  10. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuning13 Feb
  11. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
  12. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness16 Dec

Other references · 15

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  8. 2025Chain of Draft: Thinking Faster by Writing Lesscited by 297
  9. 2025Training Language Models to Reason Efficientlycited by 247
  10. 2024Distilling System 2 into System 1cited by 159
  11. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  12. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 117
3 more
  1. 2024Learning to Route LLMs with Confidence Tokenscited by 67
  2. 2024The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Modelscited by 63
  3. 2024Can Language Models Learn to Skip Steps?cited by 59

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 15.

25 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 37
  3. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 24
  4. 2025Interleaved Reasoning for Large Language Models via Reinforcement Learningcited by 21
  5. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  6. 2025Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexitycited by 9
  7. 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoningcited by 9
  8. 2026Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes Itcited by 9
  9. 2025C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoningcited by 8
  10. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  11. 2025DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chainscited by 7
  12. 2026When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoningcited by 5
13 more
  1. 2025VL Norm: Rethink Loss Aggregation in RLVRcited by 4
  2. 2026LACONIC: Length-Aware Constrained Reinforcement Learning for LLMcited by 4
  3. 2026ZAYA1-8B Technical Reportcited by 4
  4. 2026DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewardscited by 3
  5. 2026Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressurecited by 3
  6. 2026Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewardscited by 3
  7. 2026PACE: Prefix-Protected and Difficulty-Aware Compression for Efficient Reasoningcited by 2
  8. 2025LiveThinking: Enabling Real-Time Efficient Reasoning for AI-Powered Livestreaming via Reinforcement Learningcited by 1
  9. 2026Segment-Level Attribution for Selective Learning of Long Reasoning Tracescited by 1
  10. 2026CLORE: Content-Level Optimization for Reasoning Efficiencycited by 0
  11. 2026Plan Right, Then Plan Tight: Symbolic RL for Efficient Embodied Reasoningcited by 0
  12. 2026Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learningcited by 0
  13. 2026Mitigating the Length-Scaling Tax with Online Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning
  2. 2025Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards
  3. 2026LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models
  4. 2025Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penalty
  5. 2026Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning
  6. 2026To Think or Not to Think: Allocating Reasoning Where It Helps
  7. 2025Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model
  8. 2025AALC: Large Language Model Efficient Reasoning via Adaptive Accuracy-Length Control