mlsanityData to 14 Oct 2026

Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning

arXiv 2506.05256 ↗First posted 5 Jun 2025cs.AI cs.LGcited by 52

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large reasoning models (LRMs) achieve higher performance on challenging reasoning tasks by generating more tokens at inference time, but this verbosity often wastes computation on easy problems. Existing solutions, including supervised finetuning on shorter traces, user-controlled budgets, or RL with uniform penalties, either require data curation, manual configuration, or treat all problems alike regardless of difficulty. We introduce Adaptive Length Penalty (ALP), a reinforcement learning objective tailoring generation length to per-prompt solve rate. During training, ALP monitors each prompt's online solve rate through multiple rollouts and adds a differentiable penalty whose magnitude scales inversely with that rate, so confident (easy) prompts incur a high cost for extra tokens while hard prompts remain unhindered. Posttraining DeepScaleR-1.5B with ALP cuts average token usage by 50% without significantly dropping performance. Relative to fixed-budget and uniform penalty baselines, ALP redistributes its reduced budget more intelligently by cutting compute on easy prompts and reallocating saved tokens to difficult ones, delivering higher accuracy on the hardest problems with higher cost.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 17 of its 21 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

17 of 21 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning2 Apr
  2. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning6 Mar
  3. 2025s1: Simple test-time scaling31 Jan
  4. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs30 Jan
  5. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
  6. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems12 Dec
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May

Other references · 9

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  5. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  6. 2025Training Language Models to Reason Efficientlycited by 247
  7. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  8. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  9. 2024Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculumcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 37.

52 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  3. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 37
  4. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 24
  5. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  6. 2025ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shapingcited by 17
  7. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimizationcited by 16
  8. 2026On-policy Distillation with Verifiable Rewardcited by 15
  9. 2025Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Modelcited by 13
  10. 2025SABER: Switchable and Balanced Training for Efficient LLM Reasoningcited by 12
  11. 2026ATTNPO: Attention-Guided Process Supervision for Efficient Reasoningcited by 9
  12. 2025Schoenfeld's Anatomy of Mathematical Reasoning by Language Modelscited by 8
38 more
  1. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  2. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  3. 2025e1: Learning Adaptive Control of Reasoning Effortcited by 6
  4. 2026From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoningcited by 5
  5. 2026Steering Large Reasoning Models towards Concise Reasoning via Flow Matchingcited by 4
  6. 2026On the Optimal Reasoning Length for RL-Trained Language Modelscited by 4
  7. 2026ZAYA1-8B Technical Reportcited by 4
  8. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
  9. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 4
  10. 2025Verifying Meta-Awareness via Predictive Rewards in Reasoning Modelscited by 3
  11. 2025Concise Reasoning in the Lens of Lagrangian Optimizationcited by 3
  12. 2025Learning to Reason Efficiently with Discounted Reinforcement Learningcited by 3
  13. 2026ESTAR: Early-Stopping Token-Aware Reasoning For Efficient Inferencecited by 3
  14. 2026Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoningcited by 3
  15. 2026CODA: Difficulty-Aware Compute Allocation for Adaptive Reasoningcited by 3
  16. 2026DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewardscited by 3
  17. 2026Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewardscited by 3
  18. 2025Efficient Reasoning via Reward Modelcited by 2
  19. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  20. 2026Prefix Sliding for efficient test-time scalingcited by 2
  21. 2026A State-Transition Framework for Efficient LLM Reasoningcited by 1
  22. 2026Boosting LLM Reasoning via Human-Inspired Reward Shapingcited by 1
  23. 2026Length Penalties Make Chain-of-Thought Less Monitorablecited by 1
  24. 2026BAP-SQL: Budget-Aware Observation Planning for Agentic Text-to-SQLcited by 1
  25. 2025EARL: Efficient Agentic Reinforcement Learning Systems for Large Language Modelscited by 0
  26. 2026CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregationcited by 0
  27. 2026APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewardscited by 0
  28. 2026Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoningcited by 0
  29. 2026Length Value Model: Scalable Value Pretraining for Token-Level Length Modelingcited by 0
  30. 2026LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Modelscited by 0
  31. 2026CLORE: Content-Level Optimization for Reasoning Efficiencycited by 0
  32. 2026Token-Operations-Oriented Inference Optimization Techniques for Large Modelscited by 0
  33. 2026Decomposer: Learning to Decompile Symbolic Music to Programscited by 0
  34. 2026QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimizationcited by 0
  35. 2026REChart: Reasoning-Efficient Chart Editing with Large Reasoning Modelscited by 0
  36. 2026AdaThinking-E: One-Token Entropy Regulation for Adaptive Thinkingcited by 0
  37. 2026Rewarding Efficient Reasoning Improves Abstention on Underspecified Tasks in Reasoning Modelscited by 0
  38. 2026To Think or Not to Think: Allocating Reasoning Where It Helpscited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AALC: Large Language Model Efficient Reasoning via Adaptive Accuracy-Length Control
  2. 2026Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty
  3. 2025Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards
  4. 2025Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penalty
  5. 2025DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning
  6. 2025LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization
  7. 2025Shorten After You're Right: Lazy Length Penalties for Reasoning RL
  8. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization