mlsanityData to 14 Oct 2026

Do Thinking Tokens Help or Trap? Towards More Efficient Large Reasoning Model

arXiv 2506.23840 ↗First posted 30 Jun 2025cs.CL cs.AIcited by 14

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Reasoning Models (LRMs) excel at solving complex problems but face an overthinking dilemma. When handling simple tasks, they often produce verbose responses overloaded with thinking tokens (e.g., wait, however). These tokens trigger unnecessary high-level reasoning behaviors like reflection and backtracking, reducing efficiency. In this work, our pilot study reveals that these thinking-token-induced behaviors are not essential for effective problem-solving and may even hinder correct reasoning within constrained token budgets. We identify this phenomenon as the thinking trap. To mitigate this issue, we propose Dual Policy Preference Optimization (DuP-PO), a novel algorithm featuring: (1) A rollout sampling strategy that guarantees balanced exposure to responses with and without thinking tokens; (2) A fine-grained advantage control technique to dynamically regulate the prediction of target tokens; (3) A policy shaping method ensuring stable gradient contributions from thinking tokens. Experimental results on five popular math reasoning benchmarks show that DuP-PO performs well on the popular LRM, which significantly improves their token efficiency during reasoning, while achieving superior performance of the base model.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 28 of its 37 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

28 of 37 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning3 Jun
  2. 2025AdaptThink: Reasoning Models Can Learn When to Think19 May
  3. 2025Thinkless: LLM Learns When to Think19 May
  4. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL16 May
  5. 2025Dynamic Early Exit in Reasoning Models22 Apr
  6. 2025Reasoning Models Can Be Effective Without Thinking14 Apr
  7. 2025Understanding Aha Moments: from External Observations to Internal Mechanisms3 Apr
  8. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models20 Mar
  9. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuning13 Feb
  10. 2025Demystifying Long Chain-of-Thought Reasoning in LLMs5 Feb
  11. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs30 Jan
  12. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning22 Jan
1 more
  1. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness16 Dec

Other references · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  8. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  9. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  10. 2025Learning to Reason under Off-Policy Guidancecited by 281
  11. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 165
  12. 2024Distilling System 2 into System 1cited by 159
3 more
  1. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 123
  2. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 104
  3. 2025Think When You Need: Self-Adaptive Chain-of-Thought Learningcited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  2. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  3. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 12
  4. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  5. 2026DART: Draft-Agreement Routing for Training-Free Adaptive Thinking Budgets in Hybrid Reasoning Modelscited by 8
  6. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solvingcited by 6
  7. 2026TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoningcited by 6
  8. 2026Overthinking Loops in Agents: A Structural Risk via MCP Toolscited by 5
  9. 2026Stop When Further Reasoning Won't Help: Attention-State Adaptive Generation in Reasoning Modelscited by 3
  10. 2026Thinking Traps in Long Chain-of-Thought: A Measurable Study and Trap-Aware Adaptive Restartcited by 1
  11. 2026Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Modelscited by 1
  12. 2026PathCal: State-Aware Reflection-Marker Calibration for Efficient Reasoningcited by 0
2 more
  1. 2026Loose Lips: Understanding Privacy Leakage in Agentic Dialoguecited by 0
  2. 2026ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning Can Be Restored by Correcting a Few Decision Tokens
  2. 2025Don't Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models
  3. 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
  4. 2025The Price of a Second Thought: On the Evaluation of Reasoning Efficiency in Large Language Models
  5. 2026Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation
  6. 2025Not All Tokens Are What You Need In Thinking
  7. 2026Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models
  8. 2025Optimizing Anytime Reasoning via Budget Relative Policy Optimization