# Pruning the Unsurprising: Efficient LLM Reasoning via First-Token Surprisal

arXiv: https://arxiv.org/abs/2508.05988 · HTML: https://mlsanity.com/paper/2508.05988/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 8 Aug 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 25 of 25)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://mlsanity.com/paper/2502.14768/) (2025)
- [TokenSkip: Controllable Chain-of-Thought Compression in LLMs](https://mlsanity.com/paper/2502.12067/) (2025)
- [CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation](https://mlsanity.com/paper/2502.21074/) (2025)
- [Compressed Chain of Thought: Efficient Reasoning Through Dense Representations](https://mlsanity.com/paper/2412.13171/) (2024)
- [When More is Less: Understanding Chain-of-Thought Length in LLMs](https://mlsanity.com/paper/2502.07266/) (2025)
- [A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond](https://mlsanity.com/paper/2503.21614/) (2025)
- [LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs](https://mlsanity.com/paper/2504.14655/) (2025)
- [Logical Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2502.09100/) (2025)
- [Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2502.13260/) (2025)
- [From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging](https://mlsanity.com/paper/2410.01215/) (2024)
- [Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thought](https://mlsanity.com/paper/2308.08614/) (2023)
- [Uncertainty Estimation in Autoregressive Structured Prediction](https://mlsanity.com/paper/2002.07650/) (2020)
- [ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming](https://mlsanity.com/paper/2505.16667/) (2025)

## Built on by (most cited first, 30 of 32)

- [SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents](https://mlsanity.com/paper/2601.16746/) (2026)
- [LongCodeZip: Compress Long Context for Code Language Models](https://mlsanity.com/paper/2510.00446/) (2025)
- [BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens](https://mlsanity.com/paper/2508.17196/) (2025)
- [In Line with Context: Repository-Level Code Generation via Context Inlining](https://mlsanity.com/paper/2601.00376/) (2026)
- [GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts](https://mlsanity.com/paper/2601.05110/) (2026)
- [SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agents](https://mlsanity.com/paper/2606.28434/) (2026)
- [Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answering](https://mlsanity.com/paper/2601.11255/) (2026)
- [Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding](https://mlsanity.com/paper/2602.01785/) (2026)
- [Dockerless: Environment-Free Program Verifier for Coding Agents](https://mlsanity.com/paper/2606.28436/) (2026)
- [Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models](https://mlsanity.com/paper/2601.11340/) (2026)
- [Long-Chain Reasoning Distillation via Adaptive Prefix Alignment](https://mlsanity.com/paper/2601.10064/) (2026)
- [MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning](https://mlsanity.com/paper/2601.22582/) (2026)
- [SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring](https://mlsanity.com/paper/2608.09802/) (2026)
- [Do LLMs Encode Functional Importance of Reasoning Tokens?](https://mlsanity.com/paper/2601.03066/) (2026)
- [Code Is More Than Text: Uncertainty Estimation for Code Generation](https://mlsanity.com/paper/2606.09577/) (2026)
- [LLM Agents Can See Code Repositories](https://mlsanity.com/paper/2606.14061/) (2026)
- [STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability](https://mlsanity.com/paper/2606.19236/) (2026)
- [Repo0: Design-Driven Zero-to-All Code Generation](https://mlsanity.com/paper/2608.19854/) (2026)
- [Entropy-Guided Loop: Achieving Reasoning through Uncertainty-Aware Generation](https://mlsanity.com/paper/2509.00079/) (2025)
- [HyperAgent: Leveraging Hypergraphs for Topology Optimization in Multi-Agent Communication](https://mlsanity.com/paper/2510.10611/) (2025)
- [ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation](https://mlsanity.com/paper/2604.26923/) (2026)
- [SWE-Pruner Pro: The Coder LLM Already Knows What to Prune](https://mlsanity.com/paper/2607.18213/) (2026)
- [SkillForge: Self-Distilling Agents for Project-Specific Issue Resolution](https://mlsanity.com/paper/2608.18933/) (2026)
- [HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression](https://mlsanity.com/paper/2606.01934/) (2026)
- [Reasoning Denoiser: Denoising Reasoning Traces for Hallucination Detection in Large Reasoning Models](https://mlsanity.com/paper/2607.22098/) (2026)
- [ParaTempo: Efficient Parallel Reasoning via Temporal Confidence](https://mlsanity.com/paper/2608.16425/) (2026)
- [Deep Learning-based Method for Expressing Knowledge Boundary of Black-Box LLM](https://mlsanity.com/paper/2602.10801/) (2026)
- [ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction](https://mlsanity.com/paper/2604.23813/) (2026)
- [Consilience for Verifier-Free Test-Time Scaling](https://mlsanity.com/paper/2608.09898/) (2026)
- [Translator vs. Challenger: Adversarial Agentic Learning for C-to-Rust Translation](https://mlsanity.com/paper/2609.15381/) (2026)

## Abstract (arXiv)

Large Reasoning Models (LRMs) have demonstrated remarkable capabilities by scaling up the length of Chain-of-Thought (CoT). However, excessively long reasoning traces pose substantial challenges for training cost and inference latency. While various CoT compression approaches have emerged to address this challenge, they face inherent trade-offs: token-level methods often disrupt syntactic and logical coherence, while step-level methods based on perplexity fail to reliably capture the logically critical reasoning steps because of the dilution of logical information. In this paper, we propose ASAP (Anchor-guided, SurprisAl-based Pruning), a novel coarse-to-fine framework for CoT compression. ASAP first performs anchor-guided pruning to preserve the core reasoning structure, which efficiently reduces the search space for subsequent processing. Leveraging the insight that logical branching choices are concentrated at the onset of reasoning steps, it then enables logic-aware pruning by selecting logically essential reasoning steps based on a novel first-token surprisal metric. Finally, ASAP distills the models to autonomously generate and leverage these concise CoTs at inference time, enabling efficient reasoning. Experiments show that ASAP achieves state-of-the-art accuracy across multiple benchmarks while substantially reducing training and inference costs.