mlsanityData to 7 Oct 2026

Decoding Speculative Decoding

arXiv 2402.01528 ↗First posted 2 Feb 2024cs.LG cs.CLcited by 33

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative Decoding is a widely used technique to speed up inference for Large Language Models (LLMs) without sacrificing quality. When performing inference, speculative decoding uses a smaller draft model to generate speculative tokens and then uses the target LLM to verify those draft tokens. The speedup provided by speculative decoding heavily depends on the choice of the draft model. In this work, we perform a detailed study comprising over 350 experiments with LLaMA-65B and OPT-66B using speculative decoding and delineate the factors that affect the performance gain provided by speculative decoding. Our experiments indicate that the performance of speculative decoding depends heavily on the latency of the draft model, and the draft model's capability in language modeling does not correlate strongly with its performance in speculative decoding. Based on these insights we explore a new design space for draft models and design hardware-efficient draft models for speculative decoding. Our newly designed draft model can provide 111% higher throughput than existing draft models and our approach generalizes further to all LLaMA models (1/2/3.1) and supervised fine-tuned models.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 54 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

29 of 54 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  2. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models28 Oct
  3. 2023Online Speculative Decoding11 Oct
  4. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding9 Oct
  5. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  6. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  7. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  8. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2018Hierarchical Neural Story Generationcited by 1,182
  12. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
9 more
  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  2. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  3. 2022Confident Adaptive Language Modelingcited by 297
  4. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  5. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  6. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  7. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 70
  8. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  9. 2021A Runtime-Based Computational Performance Predictor for Deep Neural Network Trainingcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 16; 2026: 13.

33 papers.

Sort:

By area:

  1. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  2. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 60
  3. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  4. 2025SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decodingcited by 26
  5. 2025A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Servicescited by 15
  6. 2025Scaling Inference-Efficient Language Modelscited by 14
  7. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 14
  8. 2025Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exitscited by 8
  9. 2024FastDraft: How to Train Your Draftcited by 7
  10. 2024When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMscited by 6
  11. 2026HeiSD: Hybrid Speculative Decoding for Embodied Vision-Language-Action Models with Kinematic Awarenesscited by 6
  12. 2026DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verificationcited by 5
21 more
  1. 2025Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decodingcited by 4
  2. 2025Less is More: Improving LLM Reasoning with Minimal Test-Time Interventioncited by 4
  3. 2025HiSpec: Hierarchical Speculative Decoding for LLMscited by 3
  4. 2025What Limits Agentic Systems Efficiency?cited by 3
  5. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMscited by 3
  6. 2025Context-Enhanced Contrastive Search for Improved LLM Text Generationcited by 2
  7. 2025Confidence-Modulated Speculative Decoding for Large Language Modelscited by 1
  8. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inferencecited by 1
  9. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  10. 2026AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devicescited by 1
  11. 2026SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agentscited by 1
  12. 2026Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Offcited by 1
  13. 2026Adversarial Prompts for Acceptance Collapse in Speculative Decodingcited by 1
  14. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inferencecited by 0
  15. 2025RAGFort: Dual-Path Defense Against Proprietary Knowledge Base Extraction in Retrieval-Augmented Generationcited by 0
  16. 2026LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Losscited by 0
  17. 2026Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decodingcited by 0
  18. 2026Cross-Family Speculative Decoding for Polish Language Models on Apple~Silicon: An Empirical Evaluation of Bielik~11B with UAG-Extended MLX-LMcited by 0
  19. 2026SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inferencecited by 0
  20. 2026BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edgecited by 0
  21. 2026MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding: Performance or Illusion?
  2. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  3. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  4. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  5. 2026Speculative Decoding with a Speculative Vocabulary
  6. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models
  7. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  8. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models