mlsanityData to 7 Oct 2026

Accelerating LLM Inference with Staged Speculative Decoding

arXiv 2308.04623 ↗First posted 8 Aug 2023cs.AI cs.CLcited by 151

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Recent advances with large language models (LLM) illustrate their diverse capabilities. We propose a novel algorithm, staged speculative decoding, to accelerate LLM inference in small-batch, on-device scenarios. We address the low arithmetic intensity of small-batch inference by improving upon previous work in speculative decoding. First, we restructure the speculative batch as a tree, which reduces generation costs and increases the expected tokens per batch. Second, we add a second stage of speculative decoding. Taken together, we reduce single-batch decoding latency by 3.16x with a 762M parameter GPT-2-L model while perfectly preserving output quality.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 22 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

15 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
1 more
  1. 2022The Stack: 3 TB of permissively licensed source codecited by 354

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 71; 2025: 47; 2026: 22.

151 papers.

Sort:

By area:

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  5. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  6. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  8. 2023Efficient Large Language Models: A Surveycited by 183
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  10. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  11. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  12. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 113
38 more
  1. 2023Online Speculative Decodingcited by 109
  2. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  3. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  4. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  5. 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
  6. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  7. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  8. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  9. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
  10. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  11. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  12. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  13. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  14. 2024Octopus v2: On-device language model for super agentcited by 45
  15. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 45
  16. 2024Faster Cascades via Speculative Decodingcited by 38
  17. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 37
  18. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  19. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 35
  20. 2024A Declarative System for Optimizing AI Workloadscited by 35
  21. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  22. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanismcited by 31
  23. 2025Accelerated Diffusion Models via Speculative Samplingcited by 30
  24. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  25. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  26. 2026Accelerating Speculative Decoding with Block Diffusion Draft Treescited by 27
  27. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attributioncited by 26
  28. 2025Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decodingcited by 26
  29. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  30. 2025Efficient Reasoning for LLMs through Speculative Chain-of-Thoughtcited by 23
  31. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decodingcited by 22
  32. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  33. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decodingcited by 21
  34. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  35. 2024ZipAR: Parallel Auto-regressive Image Generation through Spatial Localitycited by 20
  36. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Modelscited by 19
  37. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
  38. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scalingcited by 19

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  2. 2025SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
  3. 2025Speculative Decoding: Performance or Illusion?
  4. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  5. 2026Faster LLM Inference via Sequential Monte Carlo
  6. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference
  7. 2025From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing
  8. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding