mlsanityData to 7 Oct 2026

Learning to Reason Efficiently with Discounted Reinforcement Learning

arXiv 2510.23486 ↗First posted 27 Oct 2025cs.LGcited by 3

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency. More broadly, in goal reaching sequential decision problems we often want to reach the goal quickly, and LRM reasoning can be viewed through this lens. We challenge the assumption that longer responses improve accuracy. By penalizing reasoning tokens using a discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell optimality in restricted policy classes, we encourage concise yet accurate reasoning, analogous to preferring shorter successful trajectories in a stochastic shortest path problem. Experiments confirm our theoretical results that this approach shortens chains of thought while preserving accuracy.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 57 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

27 of 57 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Train Long, Think Short: Curriculum Learning for Efficient Reasoning12 Aug
  2. 2025SABER: Switchable and Balanced Training for Efficient LLM Reasoning8 Aug
  3. 2025Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penalty12 Jun
  4. 2025Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning5 Jun
  5. 2025Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards23 May
  6. 2025Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoning21 May
  7. 2025ConCISE: Confidence-guided Compression in Step-by-step Efficient Reasoning8 May
  8. 2025Concise Reasoning via Reinforcement Learning7 Apr
  9. 2025Effectively Controlling Reasoning Models through Thinking Intervention31 Mar
  10. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models20 Mar
  11. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning6 Mar
  12. 2025How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approach3 Mar

Other references · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024OpenAI o1 System Cardcited by 1,220
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  8. 2024Phi-4 Technical Reportcited by 675
  9. 2025Training Language Models to Reason Efficientlycited by 242
  10. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  11. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  12. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
3 more
  1. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  2. 2025ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Modelscited by 14
  3. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Trainingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Not All Turns Are Equally Hard: Adaptive Thinking Budgets For Efficient Multi-Turn Reasoning in Agentscited by 3
  2. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  3. 2026Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
  2. 2025How Far Are We from Optimal Reasoning Efficiency?
  3. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
  4. 2025Promoting Efficient Reasoning with Verifiable Stepwise Reward
  5. 2026Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty
  6. 2025LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization
  7. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoning
  8. 2025The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning