mlsanityData to 7 Oct 2026

Concise Reasoning via Reinforcement Learning

arXiv 2504.05185 ↗First posted 7 Apr 2025cs.CLcited by 78

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

A major drawback of reasoning models is their excessive token usage, inflating computational cost, resource demand, and latency. We show this verbosity stems not from deeper reasoning but from reinforcement learning loss minimization when models produce incorrect answers. With unsolvable problems dominating training, this effect compounds into a systematic tendency toward longer outputs. Through theoretical analysis of PPO and GRPO, we prove that incorrect answers inherently drive policies toward verbosity textiteven when γ=1, reframing response lengthening as an optimization artifact. We further uncover a consistent correlation between conciseness and correctness across reasoning and non-reasoning models. Building on these insights, we propose a two-phase RL procedure where a brief secondary stage, trained on a small set of solvable problems, significantly reduces response length while preserving or improving accuracy. Finally, we show that while GRPO shares properties with PPO, it exhibits collapse modes, limiting its reliability for concise reasoning. Our claims are supported by extensive experiments.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 24 of its 42 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

24 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approach3 Mar
  2. 2025Demystifying Long Chain-of-Thought Reasoning in LLMs5 Feb
  3. 2025s1: Simple test-time scaling31 Jan
  4. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost29 Jul
  5. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards16 Apr
  6. 2024ReFT: Reasoning with Reinforced Fine-Tuning17 Jan

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  9. 2025Chain of Draft: Thinking Faster by Writing Lesscited by 290
  10. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
  11. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 182
  12. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
6 more
  1. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 95
  2. 2024The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Modelscited by 62
  3. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  4. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  5. 2023Systematic Rectification of Language Models via Dead-end Analysiscited by 16
  6. 2023DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 54; 2026: 24.

78 papers.

Sort:

By area:

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  2. 2025Learning to Reason under Off-Policy Guidancecited by 266
  3. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  4. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  5. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  6. 2025Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoningcited by 82
  7. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 70
  8. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 59
  9. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  10. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
  11. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  12. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversitycited by 41
38 more
  1. 2025Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!cited by 36
  2. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  3. 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoningcited by 29
  4. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 27
  5. 2025Interleaved Reasoning for Large Language Models via Reinforcement Learningcited by 21
  6. 2025HAPO: Training Language Models to Reason Concisely via History-Aware Policy Optimizationcited by 18
  7. 2025CME-CAD: Heterogeneous Collaborative Multi-Expert Reinforcement Learning for CAD Code Generationcited by 18
  8. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 17
  9. 2025Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learningcited by 16
  10. 2025Walk Before You Run! Concise LLM Reasoning via Reinforcement Learningcited by 15
  11. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  12. 2025Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoningcited by 14
  13. 2025ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Modelscited by 14
  14. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Modelcited by 14
  15. 2025MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancementcited by 14
  16. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalizationcited by 13
  17. 2025Chain-of-Thought Hijackingcited by 13
  18. 2025Adaptive GoGI-Skip: Coupling Goal-Gradient Importance with Dynamic Uncertainty for Efficient Reasoningcited by 12
  19. 2025KAT-V1: Kwai-AutoThink Technical Reportcited by 12
  20. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
  21. 2025Thinker: Learning to Think Fast and Slowcited by 10
  22. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimizationcited by 10
  23. 2025Understanding and Steering the Cognitive Behaviors of Reasoning Models at Test-Timecited by 10
  24. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
  25. 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoningcited by 9
  26. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 9
  27. 2025Emergent Slow Thinking in LLMs as Inverse Tree Freezingcited by 8
  28. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Modelscited by 8
  29. 2026Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVRcited by 8
  30. 2025RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMscited by 7
  31. 2025R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoningcited by 7
  32. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 7
  33. 2025Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoningcited by 6
  34. 2025Reasoning LLMs are Wandering Solution Explorerscited by 6
  35. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  36. 2025A2FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoningcited by 6
  37. 2026Mitigating Overthinking in Large Reasoning Language Models via Reasoning Path Deviation Monitoringcited by 6
  38. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoningcited by 5

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
  2. 2025Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards
  3. 2026To Think or Not to Think: Allocating Reasoning Where It Helps
  4. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning
  5. 2026It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches
  6. 2025How Far Are We from Optimal Reasoning Efficiency?
  7. 2025LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization
  8. 2026Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation