mlsanityData to 14 Oct 2026

A Theoretical Perspective for Speculative Decoding Algorithm

arXiv 2411.00841 ↗First posted 30 Oct 2024cs.LG cs.AI cs.CLcited by 29

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Transformer-based autoregressive sampling has been the major bottleneck for slowing down large language model inferences. One effective way to accelerate inference is Speculative Decoding, which employs a small model to sample a sequence of draft tokens and a large model to validate. Given its empirical effectiveness, the theoretical understanding of Speculative Decoding is falling behind. This paper tackles this gap by conceptualizing the decoding problem via markov chain abstraction and studying the key properties, output quality and inference acceleration, from a theoretical perspective. Our analysis covers the theoretical limits of speculative decoding, batch algorithms, and output quality-inference acceleration tradeoffs. Our results reveal the fundamental connections between different components of LLMs via total variation distances and show how they jointly affect the efficiency of decoding algorithms.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 48 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

39 of 48 references.

Closest ancestors · 23

Same direction, newest first

  1. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  2. 2024BASS: Batched Attention-optimized Speculative Sampling24 Apr
  3. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  4. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding10 Apr
  5. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement21 Feb
  6. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  8. 2024Decoding Speculative Decoding2 Feb
  9. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  10. 2024Multi-Candidate Speculative Decoding12 Jan
  11. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  12. 2023REST: Retrieval-Based Speculative Decoding14 Nov
11 more
  1. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models28 Oct
  2. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  3. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  4. 2023Online Speculative Decoding11 Oct
  5. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding9 Oct
  6. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  7. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  8. 2023Speculative Decoding with Big Little Decoder15 Feb
  9. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  10. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  11. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 16

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  8. 2021ViViT: A Video Vision Transformercited by 1,620
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  10. 2020A Survey on Visual Transformercited by 654
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  12. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
4 more
  1. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  2. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  3. 2024On Speculative Decoding for Multimodal Large Language Modelscited by 25
  4. 2024Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decodingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 15.

29 papers.

Sort:

By area:

  1. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  2. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithmscited by 20
  3. 2026LK Losses: Direct Acceptance Rate Optimization for Speculative Decodingcited by 18
  4. 2025Grouped Speculative Decoding for Autoregressive Image Generationcited by 17
  5. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generationcited by 16
  6. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 15
  7. 2025Speeding up Speculative Decoding via Sequential Approximate Verificationcited by 11
  8. 2025Speculative Coupled Decoding for Training-Free Lossless Acceleration of Autoregressive Visual Generationcited by 4
  9. 2025Steering Pretrained Drafters during Speculative Decodingcited by 4
  10. 2026Multi-Drafter Speculative Decoding with Alignment Feedbackcited by 4
  11. 2026DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verificationcited by 4
  12. 2025ATTS: Asynchronous Test-Time Scaling via Conformal Predictioncited by 3
17 more
  1. 2026A Practical Investigation of Training-free Relaxed Speculative Decodingcited by 3
  2. 2025SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Modelscited by 2
  3. 2026Accelerating Speculative Diffusions via Block Verificationcited by 2
  4. 2025SpecMemo: Speculative Decoding is in Your Pocketcited by 1
  5. 2025Conformal Sparsification for Bandwidth-Efficient Edge-Cloud Speculative Decodingcited by 1
  6. 2025Polybasic Speculative Decoding Through a Theoretical Perspectivecited by 1
  7. 2026Annealed Relaxation of Speculative Decoding for Faster Autoregressive Image Generationcited by 1
  8. 2026Improving the Trade-off Between Watermark Strength and Speculative Sampling Efficiency for Language Modelscited by 1
  9. 2026When Is a Draft Accepted? A Theory of Acceptance in Speculative Decodingcited by 1
  10. 2026ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shapingcited by 1
  11. 2025Speculative Decoding Speed-of-Light: Optimal Lower Bounds via Branching Random Walkscited by 0
  12. 2026CSD: Content-aware Speculative Decoding for Efficient Image Generationcited by 0
  13. 2026BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edgecited by 0
  14. 2026Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Modelscited by 0
  15. 2026Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMscited by 0
  16. 2026CAST: Cost-Aware Speculative Trees from One-Pass Block Drafterscited by 0
  17. 2026SP2ec: Adaptive Self-Speculative Decoding for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  2. 2025Speculative Decoding: Performance or Illusion?
  3. 2026Faster LLM Inference via Sequential Monte Carlo
  4. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  5. 2026Speculative Decoding with a Speculative Vocabulary
  6. 2025Speculative Sampling via Exponential Races
  7. 2026D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
  8. 2025Confidence-Modulated Speculative Decoding for Large Language Models