# Multi-Candidate Speculative Decoding

arXiv: https://arxiv.org/abs/2401.06706 · HTML: https://mlsanity.com/paper/2401.06706/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://mlsanity.com/paper/2212.12017/) (2022)
- [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023)
- [Primer: Searching for Efficient Transformers for Language Modeling](https://mlsanity.com/paper/2109.08668/) (2021)
- [SpecTr: Fast Speculative Decoding via Optimal Transport](https://mlsanity.com/paper/2310.15141/) (2023)
- [Sparse is Enough in Scaling Transformers](https://mlsanity.com/paper/2111.12763/) (2021)

## Built on by (most cited first, 30 of 31)

- [SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths](https://mlsanity.com/paper/2405.19715/) (2024)
- [Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting](https://mlsanity.com/paper/2404.18911/) (2024)
- [A Theoretical Perspective for Speculative Decoding Algorithm](https://mlsanity.com/paper/2411.00841/) (2024)
- [Towards Optimal Multi-draft Speculative Decoding](https://mlsanity.com/paper/2502.18779/) (2025)
- [Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference](https://mlsanity.com/paper/2405.18628/) (2024)
- [Accelerating Large Language Model Reasoning via Speculative Search](https://mlsanity.com/paper/2505.02865/) (2025)
- [L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models](https://mlsanity.com/paper/2505.17505/) (2025)
- [Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters](https://mlsanity.com/paper/2406.16758/) (2024)
- [Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference](https://mlsanity.com/paper/2407.09722/) (2024)
- [A Survey on Parallel Reasoning](https://mlsanity.com/paper/2510.12164/) (2025)
- [Accelerating Production LLMs with Combined Token/Embedding Speculators](https://mlsanity.com/paper/2404.19124/) (2024)
- [Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Models](https://mlsanity.com/paper/2410.20418/) (2024)
- [Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding](https://mlsanity.com/paper/2505.18629/) (2025)
- [Improving Multi-candidate Speculative Decoding](https://mlsanity.com/paper/2409.10644/) (2024)
- [Collaborative Speculative Inference for Efficient LLM Inference Serving](https://mlsanity.com/paper/2503.10325/) (2025)
- [SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences](https://mlsanity.com/paper/2505.20776/) (2025)
- [Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding](https://mlsanity.com/paper/2601.05724/) (2026)
- [Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling](https://mlsanity.com/paper/2604.04987/) (2026)
- [Multi-Drafter Speculative Decoding with Alignment Feedback](https://mlsanity.com/paper/2604.05417/) (2026)
- [AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters](https://mlsanity.com/paper/2607.19223/) (2026)
- [Reinforcement Speculative Decoding for Fast Ranking](https://mlsanity.com/paper/2505.20316/) (2025)
- [Optimized Speculative Sampling for GPU Hardware Accelerators](https://mlsanity.com/paper/2406.11016/) (2024)
- [SpecTr-GBV: Multi-Draft Block Verification Accelerating Speculative Decoding](https://mlsanity.com/paper/2604.25925/) (2026)
- [Improving the Trade-off Between Watermark Strength and Speculative Sampling Efficiency for Language Models](https://mlsanity.com/paper/2602.01428/) (2026)
- [Copy-as-Decode: Grammar-Constrained Parallel Prefill for LLM Editing](https://mlsanity.com/paper/2604.18170/) (2026)
- [Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting](https://mlsanity.com/paper/2607.10661/) (2026)
- [Cross-Attention Speculative Decoding](https://mlsanity.com/paper/2505.24544/) (2025)
- [Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple](https://mlsanity.com/paper/2603.11053/) (2026)
- [Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing](https://mlsanity.com/paper/2603.17942/) (2026)
- [Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes](https://mlsanity.com/paper/2607.26627/) (2026)

## Abstract (arXiv)

Large language models have shown impressive capabilities across a variety of NLP tasks, yet their generating text autoregressively is time-consuming. One way to speed them up is speculative decoding, which generates candidate segments (a sequence of tokens) from a fast draft model that is then verified in parallel by the target model. However, the acceptance rate of candidate tokens receives limitations from several factors, such as the model, the dataset, and the decoding setup. This paper proposes sampling multiple candidates from a draft model and then organising them in batches for verification. We design algorithms for efficient multi-candidate verification while maintaining the distribution of the target model. Our approach shows significant improvements in acceptance rates on multiple datasets and models, consistently outperforming standard speculative decoding.