mlsanityData to 7 Oct 2026

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

arXiv 2607.27735 ↗First posted 30 Jul 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost. Guided by this analysis, we introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference. SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache. It recycles per-head attention statistics produced during full-context verification as a no-extra-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache. An online entropy-based controller further selects the speculation length according to expected step-wise efficiency. Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration, with up to speedup over autoregressive decoding while preserving the target model's output distribution.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 29 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

26 of 29 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention6 Feb
  2. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding1 Dec
  3. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  4. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding27 Feb
  5. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation27 Nov
  6. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  7. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  9. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  11. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb

Other references · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  8. 2019Are Sixteen Heads Really Better than One?cited by 803
  9. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  11. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  12. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
3 more
  1. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  2. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
  2. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  3. 2026LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
  4. 2026SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
  5. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache
  6. 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
  7. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  8. 2025OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs