mlsanityData to 7 Oct 2026

Multi-Candidate Speculative Decoding

arXiv 2401.06706 ↗First posted 12 Jan 2024cs.CLcited by 30

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models have shown impressive capabilities across a variety of NLP tasks, yet their generating text autoregressively is time-consuming. One way to speed them up is speculative decoding, which generates candidate segments (a sequence of tokens) from a fast draft model that is then verified in parallel by the target model. However, the acceptance rate of candidate tokens receives limitations from several factors, such as the model, the dataset, and the decoding setup. This paper proposes sampling multiple candidates from a draft model and then organising them in batches for verification. We design algorithms for efficient multi-candidate verification while maintaining the distribution of the target model. Our approach shows significant improvements in acceptance rates on multiple datasets and models, consistently outperforming standard speculative decoding.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 27 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

19 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  2. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  12. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
3 more
  1. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  2. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  3. 2021Sparse is Enough in Scaling Transformerscited by 86

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 9; 2026: 11.

30 papers.

Sort:

By area:

  1. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  2. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  3. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  4. 2025Towards Optimal Multi-draft Speculative Decodingcited by 19
  5. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inferencecited by 16
  6. 2025Accelerating Large Language Model Reasoning via Speculative Searchcited by 16
  7. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 14
  8. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafterscited by 12
  9. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inferencecited by 11
  10. 2025A Survey on Parallel Reasoningcited by 11
  11. 2024Accelerating Production LLMs with Combined Token/Embedding Speculatorscited by 8
  12. 2024Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Modelscited by 7
18 more
  1. 2025Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decodingcited by 6
  2. 2024Improving Multi-candidate Speculative Decodingcited by 5
  3. 2025Collaborative Speculative Inference for Efficient LLM Inference Servingcited by 5
  4. 2025SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequencescited by 4
  5. 2026Overcoming Joint Intractability with Lossless Hierarchical Speculative Decodingcited by 4
  6. 2026Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Samplingcited by 4
  7. 2025Reinforcement Speculative Decoding for Fast Rankingcited by 3
  8. 2026Multi-Drafter Speculative Decoding with Alignment Feedbackcited by 3
  9. 2024Optimized Speculative Sampling for GPU Hardware Acceleratorscited by 2
  10. 2026SpecTr-GBV: Multi-Draft Block Verification Accelerating Speculative Decodingcited by 2
  11. 2026Improving the Trade-off Between Watermark Strength and Speculative Sampling Efficiency for Language Modelscited by 1
  12. 2026Copy-as-Decode: Grammar-Constrained Parallel Prefill for LLM Editingcited by 1
  13. 2026Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Draftingcited by 1
  14. 2025Cross-Attention Speculative Decodingcited by 0
  15. 2026Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simplecited by 0
  16. 2026Efficient Training-Free Multi-Token Prediction via Embedding-Space Probingcited by 0
  17. 2026Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modescited by 0
  18. 2026X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
  2. 2024SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  4. 2026D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
  5. 2023PaSS: Parallel Speculative Sampling
  6. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  7. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  8. 2026ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping