mlsanityData to 7 Oct 2026

Fast Inference from Transformers via Speculative Decoding

arXiv 2211.17192 ↗First posted 30 Nov 2022cs.LG cs.CLcited by 1,615

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Inference from large autoregressive models like Transformers is slow - decoding K tokens takes K serial runs of the model. In this work we introduce speculative decoding - an algorithm to sample from autoregressive models faster without any changes to the outputs, by computing several tokens in parallel. At the heart of our approach lie the observations that (1) hard language-modeling tasks often include easier subtasks that can be approximated well by more efficient models, and (2) using speculative execution and a novel sampling method, we can make exact decoding from the large models faster, by running them in parallel on the outputs of the approximation models, potentially generating several tokens concurrently, and without changing the distribution. Our method can accelerate existing off-the-shelf models without retraining or architecture changes. We demonstrate it on T5-XXL and show a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.

Key facts. Cited by 1,615 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 31 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

23 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding9 Jun

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  10. 2021Dynamic Neural Networks: A Surveycited by 405
  11. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  12. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
10 more
  1. 2019Depth-Adaptive Transformercited by 227
  2. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  3. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  4. 2019Adaptive Attention Span in Transformerscited by 148
  5. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  6. 2021The Efficiency Misnomercited by 98
  7. 2021Sparse is Enough in Scaling Transformerscited by 86
  8. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  9. 2020Why should we add early exits to neural networks?cited by 70
  10. 2020Controlling Computation versus Quality for Neural Sequence Modelscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 71; 2024: 340; 2025: 509; 2026: 695.

1,615 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  4. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  8. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  9. 2024Chronos: Learning the Language of Time Seriescited by 712
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  11. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  12. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
38 more
  1. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  2. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  3. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  5. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  6. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  7. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  8. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  9. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  10. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  11. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  12. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  13. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  14. 2023Large Language Models as Tool Makerscited by 293
  15. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  16. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  17. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  18. 2024Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Textcited by 255
  19. 2023Challenges and Applications of Large Language Modelscited by 251
  20. 2024Token-Budget-Aware LLM Reasoningcited by 242
  21. 2025Training Language Models to Reason Efficientlycited by 242
  22. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  23. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  24. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  25. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  26. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  27. 2026Gemma 4 Technical Reportcited by 187
  28. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  29. 2023Efficient Large Language Models: A Surveycited by 183
  30. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  31. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  32. 2023An LLM Compiler for Parallel Function Callingcited by 170
  33. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  34. 2023Speculative Decoding with Big Little Decodercited by 163
  35. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
  36. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  37. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  38. 2024Tree Search for Language Model Agentscited by 148

50 most cited of 1,615 All 1,615 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  2. 2026Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
  3. 2026Mentored Decoding: Faster Inference meets Boosting
  4. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
  5. 2024Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree
  6. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation