mlsanityData to 7 Oct 2026

Speculative Decoding and Beyond: An In-Depth Survey of Techniques

arXiv 2502.19732 ↗First posted 27 Feb 2025cs.CLcited by 22

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Sequential dependencies present a fundamental bottleneck in deploying large-scale autoregressive models, particularly for real-time applications. While traditional optimization approaches like pruning and quantization often compromise model quality, recent advances in generation-refinement frameworks demonstrate that this trade-off can be significantly mitigated. This survey presents a comprehensive taxonomy of generation-refinement frameworks, analyzing methods across autoregressive sequence tasks. We categorize methods based on their generation strategies (from simple n-gram prediction to sophisticated draft models) and refinement mechanisms (including single-pass verification and iterative approaches). Through systematic analysis of both algorithmic innovations and system-level implementations, we examine deployment strategies across computing environments and explore applications spanning text, images, and speech generation. This systematic examination of both theoretical frameworks and practical implementations provides a foundation for future research in efficient autoregressive decoding.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 99 of its 113 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

99 of 113 references.

Closest ancestors · 63

Same direction, newest first

  1. 2025Towards Optimal Multi-draft Speculative Decoding26 Feb
  2. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment31 Jan
  3. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference25 Dec
  4. 2024Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree17 Dec
  5. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration25 Nov
  6. 2024FastDraft: How to Train Your Draft17 Nov
  7. 2024SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding8 Nov
  8. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation6 Nov
  9. 2024A Theoretical Perspective for Speculative Decoding Algorithm30 Oct
  10. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding29 Oct
  11. 2024AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration22 Oct
  12. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure15 Oct
38 more
  1. 2024QSpec: Speculative Decoding with Complementary Quantization Schemes15 Oct
  2. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  3. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decoding8 Oct
  4. 2024Mixture of Attentions For Speculative Decoding4 Oct
  5. 2024Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity1 Oct
  6. 2024Dynamic-Width Speculative Beam Decoding for Efficient LLM Inference25 Sep
  7. 2024Dynamic Depth Decoding: Faster Speculative Decoding for LLMs30 Aug
  8. 2024Learning Harmonized Representations for Speculative Sampling28 Aug
  9. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  10. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  11. 2024Graph-Structured Speculative Decoding23 Jul
  12. 2024PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation16 Jul
  13. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference12 Jul
  14. 2024SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding26 Jun
  15. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure25 Jun
  16. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  17. 2024Optimized Speculative Sampling for GPU Hardware Accelerators16 Jun
  18. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism6 Jun
  19. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices4 Jun
  20. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  21. 2024Faster Cascades via Speculative Decoding29 May
  22. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting29 Apr
  23. 2024BASS: Batched Attention-optimized Speculative Sampling24 Apr
  24. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  25. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding10 Apr
  26. 2024Block Verification Accelerates Speculative Decoding15 Mar
  27. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Models14 Mar
  28. 2024CLLMs: Consistency Large Language Models28 Feb
  29. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement21 Feb
  30. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding21 Feb
  31. 2024ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding21 Feb
  32. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  33. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding19 Feb
  34. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  35. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  36. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  37. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  38. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan

50 newest of 63 All 63 (JSON)

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  9. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  10. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  11. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
24 more
  1. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  2. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  3. 2019Introduction to Multi-Armed Banditscited by 475
  4. 2021Pay Attention to MLPscited by 367
  5. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  6. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  7. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  8. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  9. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  10. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  11. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  12. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  13. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  14. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  15. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  16. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  17. 2024LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decodingcited by 46
  18. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  19. 2024On Speculative Decoding for Multimodal Large Language Modelscited by 24
  20. 2024Continuous Speculative Decoding for Autoregressive Image Generationcited by 18
  21. 2024Efficient Inference for Large Language Model-based Generative Recommendationcited by 11
  22. 2024Adaptive Draft-Verification for Efficient Large Language Model Decodingcited by 8
  23. 2025Knowledge Distillation with Training Wheelscited by 4
  24. 2024Faster Speech-LLaMA Inference with Multi-token Predictioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 13.

22 papers.

Sort:

By area:

  1. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 15
  2. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  3. 2025KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimizationcited by 10
  4. 2025Draft-based Approximate Inference for LLMscited by 6
  5. 2025Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computingcited by 5
  6. 2025Steering Pretrained Drafters during Speculative Decodingcited by 4
  7. 2026Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Samplingcited by 4
  8. 2025Accelerated Test-Time Scaling with Model-Free Speculative Samplingcited by 3
  9. 2025FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoningcited by 3
  10. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarioscited by 3
  11. 2026Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMscited by 3
  12. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 1
10 more
  1. 2026Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillationcited by 1
  2. 2026SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decodingcited by 1
  3. 2026DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generationcited by 1
  4. 2026ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shapingcited by 1
  5. 2026TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMscited by 0
  6. 2026Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devicescited by 0
  7. 2026Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generationcited by 0
  8. 2026DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Executioncited by 0
  9. 2026HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworkscited by 0
  10. 2026CAST: Cost-Aware Speculative Trees from One-Pass Block Drafterscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
  2. 2026Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks
  3. 2025Draft, Verify, and Improve: Toward Training-Aware Speculative Decoding
  4. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  5. 2026Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  7. 2025Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding
  8. 2026A Practical Investigation of Training-free Relaxed Speculative Decoding