mlsanityData to 14 Oct 2026

Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding

arXiv 2512.01278 ↗First posted 1 Dec 2025cs.LG cs.AIcited by 10

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation shifts the inference bottleneck from compute-bound to memory-bound. To generate each token, the model applies full attention to all previously generated tokens, requiring memory access to an increasingly large KV-Cache. Consequently, longer generations demand more memory access for every step, leading to substantial pressure on memory bandwidth. To address this, we introduce SparseSpec, a speculative decoding framework that reuses the same model as the draft and target models (i.e., self-speculation). SparseSpec features a novel sparse attention mechanism, PillarAttn, as the draft model, which accurately selects critical tokens via elegantly reusing information from the verification stage. Furthermore, SparseSpec co-designs self-speculation with three system innovations: (1) a unified scheduler to batch token drafting and verification, (2) delayed verification for CPU/GPU overlap, and (3) dynamic KV-Cache management to maximize memory utilization. Across various models and datasets, SparseSpec outperforms state-of-the-art solutions, with an up to 2.13x throughput speedup.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 38 of its 47 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

38 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan

Other references · 36

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  6. 2024OpenAI o1 System Cardcited by 1,244
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
24 more
  1. 2024DeepSeek-V3 Technical Reportcited by 811
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  4. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  5. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  6. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  7. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  8. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  9. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  10. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 117
  11. 2023Online Speculative Decodingcited by 113
  12. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  13. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  14. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 91
  15. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  16. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  17. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  18. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
  19. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 39
  20. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 38
  21. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 36
  22. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanismcited by 32
  23. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMscited by 24
  24. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 16
  2. 2026Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rolloutscited by 9
  3. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 7
  4. 2026DFlow: Enabling Verifier Information Flow in Block Diffusion Speculative Decodingcited by 2
  5. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decodingcited by 1
  6. 2026ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inferencecited by 1
  7. 2026TreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative Decodingcited by 1
  8. 2026Self-Speculation for Faster Reasoning Modelscited by 0
  9. 2026Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decodingcited by 0
  10. 2026S3R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SpecExit: Accelerating Large Reasoning Model via Speculative Exit
  2. 2025SpecAttn: Speculating Sparse Attention
  3. 2026KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem
  4. 2024SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding
  5. 2026ConfSpec: Efficient Step-Level Speculative Reasoning via Confidence-Gated Verification
  6. 2025Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention
  7. 2026From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning
  8. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning