On the Optimal Reasoning Length for RL-Trained Language Models
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Reinforcement learning substantially improves reasoning in large language models, but it also tends to lengthen chain-of-thought outputs and increase computational cost. Although length-control methods have been proposed, the length-accuracy relationship they induce remains unclear. We train policies with several length-control methods on multiple base models in a controlled setup and find that, across both mathematical reasoning and code generation, accuracy is non-monotonic in output length, peaking at an intermediate value. Mode accuracy, however, continues to improve with length even in settings where sample accuracy plateaus or declines, indicating that the non-monotonic length-accuracy relationship is driven by dispersion around an increasingly correct center.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 11 of its 18 references that are in the map. First posted on arXiv 10 Feb 2026.
Builds on
11 of 18 references.
Closest ancestors · 3
Same direction, newest first
- 2025Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning5 Jun
- 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs30 Apr
- 2025Competitive Programming with Large Reasoning Models3 Feb
Other references · 8
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2024OpenAI o1 System Cardcited by 1,220
- 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
- 2025Defeating the Training-Inference Mismatch via FP16cited by 62
- 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimizationcited by 16
- 2025Eliminating Hallucination-Induced Errors in LLM Code Generation with Functional Clusteringcited by 2
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Implicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Trainingcited by 1
- 2026DRT: Dense Reasoning Trace for Efficient and Grounded Multimodal Reasoningcited by 0
- 2026Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learningcited by 0
- 2026Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learning28 Sep 2026
- 2026DRT: Dense Reasoning Trace for Efficient and Grounded Multimodal Reasoning18 Sep 2026
- 2026Implicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Training8 May 2026
Similar, not cited
Close in wording, no citation either way
- 2025Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards
- 2026To Think or Not to Think: Allocating Reasoning Where It Helps
- 2025AALC: Large Language Model Efficient Reasoning via Adaptive Accuracy-Length Control
- 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
- 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
- 2025When More is Less: Understanding Chain-of-Thought Length in LLMs
- 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
- 2025Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penalty