# Think Clearly: Improving Reasoning via Redundant Token Pruning

arXiv: https://arxiv.org/abs/2507.08806 · HTML: https://mlsanity.com/paper/2507.08806/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Efficient LLM inference and serving / [Efficient reasoning and agent inference](https://mlsanity.com/inference/efficient-reasoning-and-agent-inference/)

## Builds on (most cited first, 20 of 20)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [Chain of Draft: Thinking Faster by Writing Less](https://mlsanity.com/paper/2502.18600/) (2025)
- [Why think step by step? Reasoning emerges from the locality of experience](https://mlsanity.com/paper/2304.03843/) (2023)
- [Faithful Logical Reasoning via Symbolic Chain-of-Thought](https://mlsanity.com/paper/2405.18357/) (2024)
- [Effectively Controlling Reasoning Models through Thinking Intervention](https://mlsanity.com/paper/2503.24370/) (2025)
- [Offline Reinforcement Learning for LLM Multi-Step Reasoning](https://mlsanity.com/paper/2412.16145/) (2024)
- [Multi-hop Question Answering via Reasoning Chains](https://mlsanity.com/paper/1910.02610/) (2019)
- [ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification](https://mlsanity.com/paper/2502.14565/) (2025)
- [Break the Chain: Large Language Models Can be Shortcut Reasoners](https://mlsanity.com/paper/2406.06580/) (2024)
- [The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking](https://mlsanity.com/paper/2501.19358/) (2025)

## Built on by (most cited first, 17 of 17)

- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning](https://mlsanity.com/paper/2602.09953/) (2026)
- [GLARE: Agentic Reasoning for Legal Judgment Prediction](https://mlsanity.com/paper/2508.16383/) (2025)
- [FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning](https://mlsanity.com/paper/2601.19001/) (2026)
- [Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression](https://mlsanity.com/paper/2510.01581/) (2025)
- [interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification](https://mlsanity.com/paper/2602.11202/) (2026)
- [Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution](https://mlsanity.com/paper/2602.16154/) (2026)
- [Practical Online KV Cache Compaction for LLM Agents: An Empirical Study](https://mlsanity.com/paper/2608.00902/) (2026)
- [Dynamic Thinking-Token Selection for Efficient Reasoning in Large Reasoning Models](https://mlsanity.com/paper/2601.18383/) (2026)
- [CRISP: Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency Pruning](https://mlsanity.com/paper/2604.17297/) (2026)
- [VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning](https://mlsanity.com/paper/2605.31457/) (2026)
- [VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervision](https://mlsanity.com/paper/2510.27462/) (2025)
- [Calibrating Transformer Attention via Task-Space Sensitivity Feedback](https://mlsanity.com/paper/2512.20661/) (2025)
- [Beyond Model Scaling: Test-Time Intervention for Efficient Deep Reasoning](https://mlsanity.com/paper/2601.11252/) (2026)
- [Graph-Based Chain-of-Thought Pruning for Reducing Redundant Reflections in Reasoning LLMs](https://mlsanity.com/paper/2604.05643/) (2026)
- [State commitment learning: training language models to distinguish computation from memory](https://mlsanity.com/paper/2606.05201/) (2026)

## Abstract (arXiv)

Recent large language models have shown promising capabilities in long-form reasoning, following structured chains of thought before arriving at a final answer. However, we observe that these reasoning paths tend to include substantial redundancy; analyzing attention patterns reveals that attention scores are widely scattered, particularly incorrect answers exhibit greater attention sparsity. In this paper, we demonstrate that deliberately removing this redundancy in the reasoning process significantly improves performance through clear thinking, i.e., removing distraction. Specifically, we systematically identify reasoning redundancy by measuring token-level attention scores to a special end-of-thinking token, which is appended to an explicit instruction inserted to conclude each intermediate reasoning step. Furthermore, we propose structure-aware pruning that prioritizes removing tokens in low-contributing reasoning chunks over individual tokens. After evicting redundant tokens, we remove the injected end-of-thinking instruction, then resume the reasoning generation. We demonstrate that our method significantly improves overall accuracy across reasoning-intensive benchmarks without any training involved. In particular, our method shows strong performance on challenging mathematical competition benchmarks such as AIME and AMC, where reasoning redundancy is more prevalent.