mlsanityData to 7 Oct 2026

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding

arXiv 2401.07851 ↗First posted 15 Jan 2024cs.CLcited by 266

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

To mitigate the high inference latency stemming from autoregressive decoding in Large Language Models (LLMs), Speculative Decoding has emerged as a novel decoding paradigm for LLM inference. In each decoding step, this method first drafts several future tokens efficiently and then verifies them in parallel. Unlike autoregressive decoding, Speculative Decoding facilitates the simultaneous decoding of multiple tokens per step, thereby accelerating inference. This paper presents a comprehensive overview and analysis of this promising decoding paradigm. We begin by providing a formal definition and formulation of Speculative Decoding. Then, we organize in-depth discussions on its key facets, such as drafter selection and verification strategies. Furthermore, we present a comparative analysis of leading methods under third-party testing environments. We aim for this work to serve as a catalyst for further research on Speculative Decoding, ultimately contributing to more efficient LLM inference.

Key facts. Cited by 266 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 60 references that are in the map. First posted on arXiv 15 Jan 2024.

Builds on

36 of 60 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  2. 2023PaSS: Parallel Speculative Sampling22 Nov
  3. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  4. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  5. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  6. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2023Speculative Decoding with Big Little Decoder15 Feb
  9. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  10. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  11. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar
  12. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding9 Jun

Other references · 24

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  12. 2016Sequence-Level Knowledge Distillationcited by 761
12 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  2. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  3. 2017Non-Autoregressive Neural Machine Translationcited by 457
  4. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  5. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
  6. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  7. 2021Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 57
  8. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  9. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  10. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  11. 2022Lossless Acceleration for Seq2seq Generation with Aggressive Decodingcited by 18
  12. 2022ngram-OAXE: Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 71; 2025: 102; 2026: 93.

266 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  3. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  6. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  8. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 113
  9. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  10. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  11. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  12. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
38 more
  1. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  2. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
  3. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  4. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  5. 2025DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressivelycited by 56
  6. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  7. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 45
  8. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  9. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  10. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  11. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  12. 2024Faster Cascades via Speculative Decodingcited by 38
  13. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  14. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  15. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 34
  16. 2024CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Followingcited by 32
  17. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 31
  18. 2024Block Verification Accelerates Speculative Decodingcited by 30
  19. 2025Accelerated Diffusion Models via Speculative Samplingcited by 30
  20. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  21. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  22. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacementcited by 27
  23. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  24. 2025VeriThinker: Learning to Verify Makes Reasoning Model Efficientcited by 24
  25. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  26. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  27. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 21
  28. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  29. 2024Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Treecited by 20
  30. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  31. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  32. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 18
  33. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  34. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithmscited by 18
  35. 2024Mixture of Attentions For Speculative Decodingcited by 17
  36. 2024EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generationcited by 17
  37. 2024PLD+: Accelerating LLM inference by leveraging Language Model Artifactscited by 16
  38. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16

50 most cited of 266 All 266 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Decoding Speculative Decoding
  2. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  3. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  4. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models
  5. 2026Faster LLM Inference via Sequential Monte Carlo
  6. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
  7. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies
  8. 2026Speculative Decoding and the Curse of Multilinguality