mlsanityData to 14 Oct 2026

SPIRe: Boosting LLM Inference Throughput with Speculative Decoding

arXiv 2504.06419 ↗First posted 8 Apr 2025cs.LGcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD) has been shown to reduce the latency of autoregressive decoding (AD) by 2-3x for small batch sizes. However, increasing throughput and therefore reducing the cost per token requires decoding with large batch sizes. Recent work shows that SD can accelerate decoding with large batch sizes too if the context is sufficiently long and the draft model's KV cache is sparse. We introduce SPIRe, a draft model that combines static sparse attention, pruned initialization, and feedback memory to increase the modeled throughput of speculative decoding by over 100% compared to speculation with a much smaller draft model and by over 35% compared to the strong baseline of sparse self-speculation. Our approach is particularly effective when context lengths vary significantly across requests.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 10 of its 19 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

10 of 19 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  2. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  4. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  5. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  6. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 4

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2022Efficiently Scaling Transformer Inferencecited by 539

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AgentSpec: Speculative Decoding for Batch Inference of LLM Agentscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Accelerating LLM Inference with Staged Speculative Decoding
  2. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  3. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  4. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
  5. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  6. 2025Speculative Decoding: Performance or Illusion?
  7. 2026ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
  8. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference