mlsanityData to 7 Oct 2026

L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning

arXiv 2503.04697 ↗First posted 6 Mar 2025cs.CL cs.AI cs.LGcited by 381

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Reasoning language models have shown an uncanny ability to improve performance at test-time by ``thinking longer''-that is, by generating longer chain-of-thought sequences and hence using more compute. However, the length of their chain-of-thought reasoning is not controllable, making it impossible to allocate test-time compute to achieve a desired level of performance. We introduce Length Controlled Policy Optimization (LCPO), a simple reinforcement learning method that optimizes for accuracy and adherence to user-specified length constraints. We use LCPO to train L1, a reasoning language model that produces outputs satisfying a length constraint given in its prompt. L1's length control allows for smoothly trading off computational cost and accuracy on a wide range of tasks, and outperforms the state-of-the-art S1 method for length control. Furthermore, we uncover an unexpected short chain-of-thought capability in models trained with LCPO. Specifically, using LCPO we derive Short Reasoning Models (SRMs), that exhibit similar reasoning patterns as full-length reasoning models, but can generate CoT lengths comparable to non-reasoning models. They demonstrate significant performance gains, for instance, our 1.5B L1 model surpasses GPT-4o at equal reasoning lengths. Overall, LCPO enables precise control over reasoning length, allowing for fine-grained allocation of test-time compute and accuracy. We release code and models at https://www.cmu-l3.github.io/l1

Key facts. Cited by 381 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 24 of its 29 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

24 of 29 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025s1: Simple test-time scaling31 Jan
  2. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs30 Jan
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
  4. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness16 Dec
  5. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems12 Dec
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  7. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs19 May
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  9. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 15

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  9. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  10. 2022Generating Sequences by Learning to Self-Correctcited by 271
  11. 2025Training Language Models to Reason Efficientlycited by 242
  12. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
3 more
  1. 2024Following Length Constraints in Instructionscited by 45
  2. 2023Prompt-Based Length Controlled Generation with Reinforcement Learningcited by 14
  3. 2024Precise Length Control in Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 228; 2026: 153.

381 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  4. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  5. 2025Reasoning Models Can Be Effective Without Thinkingcited by 182
  6. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  7. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  8. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
  9. 2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 141
  10. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  11. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  12. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMscited by 115
38 more
  1. 2025Thinkless: LLM Learns When to Thinkcited by 107
  2. 2025Efficient Reasoning Models: A Surveycited by 93
  3. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 91
  4. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  5. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  6. 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learningcited by 86
  7. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80
  8. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  9. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  10. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 70
  11. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  12. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  13. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 59
  14. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  15. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  16. 2025Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?cited by 58
  17. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoningcited by 57
  18. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  19. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 53
  20. 2025Reinforcement Learning from Human Feedbackcited by 53
  21. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
  22. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 52
  23. 2025Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learningcited by 51
  24. 2025AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgetingcited by 50
  25. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  26. 2025Inverse Scaling in Test-Time Computecited by 49
  27. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 49
  28. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoningcited by 48
  29. 2025SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoningcited by 47
  30. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  31. 2025m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning with Large Language Modelscited by 42
  32. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Spacecited by 40
  33. 2025ARM: Adaptive Reasoning Modelcited by 38
  34. 2025Think When You Need: Self-Adaptive Chain-of-Thought Learningcited by 37
  35. 2025Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Modelscited by 37
  36. 2025Optimizing Length Compression in Large Reasoning Modelscited by 37
  37. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 36
  38. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35

50 most cited of 381 All 381 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Optimal Reasoning Length for RL-Trained Language Models
  2. 2025Evaluating the Safety and Skill Reasoning of Large Reasoning Models Under Compute Constraints
  3. 2025Adaptive Deep Reasoning: Triggering Deep Thinking When Needed
  4. 2025When More is Less: Understanding Chain-of-Thought Length in LLMs
  5. 2025Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning
  6. 2026Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR
  7. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning
  8. 2025SmartThinker: Learning to Compress and Preserve Reasoning by Step-Level Length Control