mlsanityData to 7 Oct 2026

Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation

arXiv 2203.16487 ↗First posted 30 Mar 2022cs.CL cs.LGcited by 169

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spec-Drafter -- an independent model specially optimized for efficient and accurate drafting -- and Spec-Verification -- a reliable method for verifying the drafted tokens efficiently in the decoding paradigm. Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around 5× speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only 1.4×~2× speedup. In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications. Our models and codes are available at https://github.com/hemingkx/SpecDec.

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 44 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

24 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding9 Jun

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2015Teaching Machines to Read and Comprehendcited by 1,039
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  7. 2016Sequence-Level Knowledge Distillationcited by 761
  8. 2017Non-Autoregressive Neural Machine Translationcited by 457
  9. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  10. 2018Scaling Neural Machine Translationcited by 204
  11. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  12. 2020GECToR -- Grammatical Error Correction: Tag, Not Rewritecited by 177
11 more
  1. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  2. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  3. 2019Encode, Tag, Realize: High-Precision Text Editingcited by 99
  4. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  5. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  6. 2018End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classificationcited by 72
  7. 2020Aligned Cross Entropy for Non-Autoregressive Machine Translationcited by 65
  8. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  9. 2021Non-Autoregressive Translation by Learning Target Categorical Codescited by 25
  10. 2020Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correctioncited by 24
  11. 2021Duplex Sequence-to-Sequence Learning for Reversible Machine Translationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 50; 2025: 63; 2026: 55.

169 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  3. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  4. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  5. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  6. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  7. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  8. 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
  9. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  10. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  11. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
  12. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
38 more
  1. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  2. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  3. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  4. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  5. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 35
  6. 2025Plan for Speed: Dilated Scheduling for Masked Diffusion Language Modelscited by 34
  7. 2024Decoding Speculative Decodingcited by 33
  8. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptancecited by 31
  9. 2025Accelerated Diffusion Models via Speculative Samplingcited by 30
  10. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  11. 2025Next Block Prediction: Video Generation via Semi-Autoregressive Modelingcited by 26
  12. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  13. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  14. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  15. 2024RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generationcited by 20
  16. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  17. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  18. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 19
  19. 2025Sequential Diffusion Language Modelscited by 19
  20. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  21. 2024Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matchingcited by 18
  22. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  23. 2024Mixture of Attentions For Speculative Decodingcited by 17
  24. 2024A Survey on Hardware Accelerators for Large Language Modelscited by 15
  25. 2025Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devicescited by 15
  26. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 15
  27. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Accelerationcited by 14
  28. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13
  29. 2024CHAI: Clustered Head Attention for Efficient LLM Inferencecited by 13
  30. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  31. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculationcited by 13
  32. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  33. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  34. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  35. 2025Parallel Sampling from Masked Diffusion Models via Conditional Independence Testingcited by 13
  36. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
  37. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 12
  38. 2025Fast Large Language Model Collaborative Decoding via Speculationcited by 12

50 most cited of 169 All 169 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Speculative Speculative Decoding
  2. 2026Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution
  3. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths
  4. 2025DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting
  5. 2026DLoop: Looped Speculative Decoding
  6. 2026SpecTr-GBV: Multi-Draft Block Verification Accelerating Speculative Decoding
  7. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  8. 2023SpecTr: Fast Speculative Decoding via Optimal Transport