mlsanityData to 7 Oct 2026

Training Language Models to Reason Efficiently

arXiv 2502.04463 ↗First posted 6 Feb 2025cs.LG cs.CLcited by 242

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Scaling model size and training data has led to great advances in the performance of Large Language Models (LLMs). However, the diminishing returns of this approach necessitate alternative methods to improve model capabilities, particularly in tasks requiring advanced reasoning. Large reasoning models, which leverage long chain-of-thoughts, bring unprecedented breakthroughs in problem-solving capabilities but at a substantial deployment cost associated to longer generations. Reducing inference costs is crucial for the economic feasibility, user experience, and environmental sustainability of these models. In this work, we propose to train large reasoning models to reason efficiently. More precisely, we use reinforcement learning (RL) to train reasoning models to dynamically allocate inference-time compute based on task complexity. Our method incentivizes models to minimize unnecessary computational overhead while maintaining accuracy, thereby achieving substantial efficiency gains. It enables the derivation of a family of reasoning models with varying efficiency levels, controlled via a single hyperparameter. Experiments on two open-weight large reasoning models demonstrate significant reductions in inference cost while preserving most of the accuracy.

Key facts. Cited by 242 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 53 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

30 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Let's Verify Step by Step31 May
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
15 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  3. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  4. 2018Rethinking the Value of Network Pruningcited by 595
  5. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  6. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  7. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 278
  8. 2022Generating Sequences by Learning to Self-Correctcited by 271
  9. 2024Reinforcement learningcited by 236
  10. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  11. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  12. 2024Stream of Search (SoS): Learning to Search in Languagecited by 141
  13. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117
  14. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 108
  15. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 69

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 149; 2026: 93.

242 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  5. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  6. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  7. 2025Reasoning Models Can Be Effective Without Thinkingcited by 182
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  9. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  10. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Modelscited by 154
  11. 2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 141
  12. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
38 more
  1. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
  2. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 116
  3. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  4. 2025Efficient Reasoning Models: A Surveycited by 93
  5. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  6. 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learningcited by 86
  7. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  8. 2025Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoningcited by 82
  9. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  10. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 70
  11. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 69
  12. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  13. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 53
  14. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
  15. 2025Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learningcited by 51
  16. 2025AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgetingcited by 50
  17. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  18. 2025Inverse Scaling in Test-Time Computecited by 49
  19. 2025SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoningcited by 47
  20. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  21. 2025ARM: Adaptive Reasoning Modelcited by 38
  22. 2025Optimizing Length Compression in Large Reasoning Modelscited by 37
  23. 2025Cost-of-Pass: An Economic Framework for Evaluating Language Modelscited by 35
  24. 2025OptimalThinkingBench: Evaluating Over and Underthinking in LLMscited by 35
  25. 2025ConCISE: Confidence-guided Compression in Step-by-step Efficient Reasoningcited by 33
  26. 2025Scalable Chain of Thoughts via Elastic Reasoningcited by 32
  27. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  28. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  29. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 29
  30. 2025Optimizing Anytime Reasoning via Budget Relative Policy Optimizationcited by 27
  31. 2025Stable Reinforcement Learning for Efficient Reasoningcited by 27
  32. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 27
  33. 2025Promoting Efficient Reasoning with Verifiable Stepwise Rewardcited by 27
  34. 2025BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokenscited by 27
  35. 2025When to Continue Thinking: Adaptive Thinking Mode Switching for Efficient Reasoningcited by 26
  36. 2025Shorten After You're Right: Lazy Length Penalties for Reasoning RLcited by 25
  37. 2025VeriThinker: Learning to Verify Makes Reasoning Model Efficientcited by 24
  38. 2025Position: Agent Should Invoke External Tools ONLY When Epistemically Necessarycited by 22

50 most cited of 242 All 242 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
  2. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models
  3. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  4. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  5. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  6. 2026Training Large Reasoning Models Efficiently via Progressive Thought Encoding
  7. 2025Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuning
  8. 2025Interleaved Reasoning for Large Language Models via Reinforcement Learning