# Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model

arXiv: https://arxiv.org/abs/2512.21540 · HTML: https://mlsanity.com/paper/2512.21540/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 27 of its 31 references that are in the map.
- First posted on arXiv 25 Dec 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 27 of 27)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://mlsanity.com/paper/2502.14768/) (2025)
- [Training Language Models to Reason Efficiently](https://mlsanity.com/paper/2502.04463/) (2025)
- [Dynamic Early Exit in Reasoning Models](https://mlsanity.com/paper/2504.15895/) (2025)
- [ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning](https://mlsanity.com/paper/2504.01296/) (2025)
- [Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions](https://mlsanity.com/paper/2506.07527/) (2025)
- [Efficient Reasoning Models: A Survey](https://mlsanity.com/paper/2504.10903/) (2025)
- [Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning](https://mlsanity.com/paper/2505.17813/) (2025)
- [Learn to Reason Efficiently with Adaptive Length-based Reward Shaping](https://mlsanity.com/paper/2505.15612/) (2025)
- [Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning](https://mlsanity.com/paper/2508.09726/) (2025)
- [Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning](https://mlsanity.com/paper/2506.05256/) (2025)
- [Reasoning to Learn from Latent Thoughts](https://mlsanity.com/paper/2503.18866/) (2025)
- [Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards](https://mlsanity.com/paper/2505.18298/) (2025)

## Built on by (most cited first, 13 of 13)

- [CRISP: Compressed Reasoning via Iterative Self-Policy Distillation](https://mlsanity.com/paper/2603.05433/) (2026)
- [Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoning](https://mlsanity.com/paper/2602.18232/) (2026)
- [Contrastive On-Policy Distillation](https://mlsanity.com/paper/2607.19046/) (2026)
- [ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression](https://mlsanity.com/paper/2605.07501/) (2026)
- [Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning](https://mlsanity.com/paper/2606.21943/) (2026)
- [Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning](https://mlsanity.com/paper/2607.15736/) (2026)
- [APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards](https://mlsanity.com/paper/2602.00760/) (2026)
- [Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling](https://mlsanity.com/paper/2604.27039/) (2026)
- [BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responses](https://mlsanity.com/paper/2605.28028/) (2026)
- [QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization](https://mlsanity.com/paper/2607.21793/) (2026)
- [Video-FLAIR: Not Whether to Reason, But How](https://mlsanity.com/paper/2608.26495/) (2026)
- [Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learning](https://mlsanity.com/paper/2609.34718/) (2026)
- [Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation](https://mlsanity.com/paper/2610.09587/) (2026)

## Abstract (arXiv)

Existing approaches typically rely on fixed length penalties, but such penalties are hard to tune and fail to adapt to the evolving reasoning abilities of LLMs, leading to suboptimal trade-offs between accuracy and conciseness. To address this challenge, we propose Leash (adaptive LEngth penAlty and reward SHaping), a reinforcement learning framework for efficient reasoning in LLMs. We formulate length control as a constrained optimization problem and employ a Lagrangian primal-dual method to dynamically adjust the penalty coefficient. When generations exceed the target length, the penalty is intensified; when they are shorter, it is relaxed. This adaptive mechanism guides models toward producing concise reasoning without sacrificing task performance. Experiments on Deepseek-R1-Distill-Qwen-1.5B and Qwen3-4B-Thinking-2507 show that Leash reduces the average reasoning length by 60% across diverse tasks - including in-distribution mathematical reasoning and out-of-distribution domains such as coding and instruction following - while maintaining competitive performance. Our work thus presents a practical and effective paradigm for developing controllable and efficient LLMs that balance reasoning capabilities with computational budgets.