mlsanityData to 7 Oct 2026

Accelerating Large Language Model Decoding with Speculative Sampling

arXiv 2302.01318 ↗First posted 2 Feb 2023cs.CLcited by 975

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the generation of multiple tokens from each transformer call. Our algorithm relies on the observation that the latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comparable to that of sampling a single token from the larger target model. This is combined with a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics. We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model, achieving a 2-2.5x decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself.

Key facts. Cited by 975 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 22 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

20 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2021ViViT: A Video Vision Transformercited by 1,610
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
7 more
  1. 2016Sequence-Level Knowledge Distillationcited by 761
  2. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  3. 2022Efficiently Scaling Transformer Inferencecited by 529
  4. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  5. 2020Accelerating Feedforward Computation via Parallel Nonlinear Equation Solvingcited by 51
  6. 2022Lossless Acceleration for Seq2seq Generation with Aggressive Decodingcited by 18
  7. 2020Predictive Sampling with Forecasting Autoregressive Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 54; 2024: 222; 2025: 295; 2026: 404.

975 papers.

Sort:

By area:

  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  4. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  5. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  6. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  7. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  8. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  9. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  10. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  11. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  12. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
38 more
  1. 2023Large Language Models as Tool Makerscited by 293
  2. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  3. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  4. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  5. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  6. 2024Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Textcited by 255
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  9. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  10. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  11. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  12. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  13. 2023Efficient Large Language Models: A Surveycited by 183
  14. 2023An LLM Compiler for Parallel Function Callingcited by 170
  15. 2023Speculative Decoding with Big Little Decodercited by 163
  16. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
  17. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  18. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  19. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  20. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  21. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  22. 2023Controlled Decoding from Language Modelscited by 133
  23. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  24. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  25. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  26. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  27. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  28. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  29. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  30. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 113
  31. 2023Parallel Sampling of Diffusion Modelscited by 112
  32. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  33. 2023Online Speculative Decodingcited by 109
  34. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  35. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  36. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  37. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  38. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103

50 most cited of 975 All 975 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
  2. 2026LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization
  3. 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
  4. 2026A Practical Investigation of Training-free Relaxed Speculative Decoding
  5. 2026MineDraft: A Framework for Batch Parallel Speculative Decoding
  6. 2025From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing
  7. 2024AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability
  8. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation