mlsanityData to 14 Oct 2026

Pruning the Unsurprising: Efficient LLM Reasoning via First-Token Surprisal

arXiv 2508.05988 ↗First posted 8 Aug 2025cs.LG cs.SEcited by 32

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Reasoning Models (LRMs) have demonstrated remarkable capabilities by scaling up the length of Chain-of-Thought (CoT). However, excessively long reasoning traces pose substantial challenges for training cost and inference latency. While various CoT compression approaches have emerged to address this challenge, they face inherent trade-offs: token-level methods often disrupt syntactic and logical coherence, while step-level methods based on perplexity fail to reliably capture the logically critical reasoning steps because of the dilution of logical information. In this paper, we propose ASAP (Anchor-guided, SurprisAl-based Pruning), a novel coarse-to-fine framework for CoT compression. ASAP first performs anchor-guided pruning to preserve the core reasoning structure, which efficiently reduces the search space for subsequent processing. Leveraging the insight that logical branching choices are concentrated at the onset of reasoning steps, it then enables logic-aware pruning by selecting logically essential reasoning steps based on a novel first-token surprisal metric. Finally, ASAP distills the models to autonomously generate and leverage these concise CoTs at inference time, enabling efficient reasoning. Experiments show that ASAP achieves state-of-the-art accuracy across multiple benchmarks while substantially reducing training and inference costs.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 25 of its 48 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

25 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond27 Mar
  2. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMs17 Feb
  3. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representations17 Dec

Other references · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2024OpenAI o1 System Cardcited by 1,244
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  10. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  11. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  12. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
10 more
  1. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  2. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 225
  3. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 183
  4. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMscited by 70
  5. 2025Logical Reasoning in Large Language Models: A Surveycited by 55
  6. 2025Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Modelscited by 51
  7. 2024From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debuggingcited by 46
  8. 2023Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thoughtcited by 41
  9. 2020Uncertainty Estimation in Autoregressive Structured Predictioncited by 23
  10. 2025ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programmingcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 28.

32 papers.

Sort:

By area:

  1. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 44
  2. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 39
  3. 2025BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokenscited by 29
  4. 2026In Line with Context: Repository-Level Code Generation via Context Inliningcited by 18
  5. 2026GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughtscited by 17
  6. 2026SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agentscited by 15
  7. 2026Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answeringcited by 14
  8. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  9. 2026Dockerless: Environment-Free Program Verifier for Coding Agentscited by 9
  10. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  11. 2026Long-Chain Reasoning Distillation via Adaptive Prefix Alignmentcited by 6
  12. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learningcited by 6
20 more
  1. 2026SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoringcited by 6
  2. 2026Do LLMs Encode Functional Importance of Reasoning Tokens?cited by 5
  3. 2026Code Is More Than Text: Uncertainty Estimation for Code Generationcited by 4
  4. 2026LLM Agents Can See Code Repositoriescited by 3
  5. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3
  6. 2026Repo0: Design-Driven Zero-to-All Code Generationcited by 3
  7. 2025Entropy-Guided Loop: Achieving Reasoning through Uncertainty-Aware Generationcited by 2
  8. 2025HyperAgent: Leveraging Hypergraphs for Topology Optimization in Multi-Agent Communicationcited by 2
  9. 2026ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generationcited by 2
  10. 2026SWE-Pruner Pro: The Coder LLM Already Knows What to Prunecited by 2
  11. 2026SkillForge: Self-Distilling Agents for Project-Specific Issue Resolutioncited by 2
  12. 2026HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compressioncited by 1
  13. 2026Reasoning Denoiser: Denoising Reasoning Traces for Hallucination Detection in Large Reasoning Modelscited by 1
  14. 2026ParaTempo: Efficient Parallel Reasoning via Temporal Confidencecited by 1
  15. 2026Deep Learning-based Method for Expressing Knowledge Boundary of Black-Box LLMcited by 0
  16. 2026ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstructioncited by 0
  17. 2026Consilience for Verifier-Free Test-Time Scalingcited by 0
  18. 2026Translator vs. Challenger: Adversarial Agentic Learning for C-to-Rust Translationcited by 0
  19. 2026Schr"odinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?cited by 0
  20. 2026TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoning
  2. 2026CtrlCoT: Dual-Granularity Chain-of-Thought Compression for Controllable Reasoning
  3. 2026Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression
  4. 2025Can Reasoning Path still be Effective as Input? Bridging Post-Reasoning to Chain-of-Thought Compression
  5. 2026Think Less, Know More: State-Aware Reasoning Compression with Knowledge Guidance for Efficient Reasoning
  6. 2026Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling
  7. 2025R1-Compress: Long Chain-of-Thought Compression via Chunk Compression and Search
  8. 2026Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models