mlsanityData to 14 Oct 2026

Cascade Speculative Drafting for Even Faster LLM Inference

arXiv 2312.11462 ↗First posted 18 Dec 2023cs.LG cs.CLcited by 83

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Introduced to enhance the efficiency of large language model (LLM) inference, speculative decoding operates by having a smaller model generate a draft. A larger target model then reviews this draft to align with its output, and any acceptance by the target model results in a reduction of the number of the target model runs, ultimately improving efficiency. However, the drafting process in speculative decoding includes slow autoregressive generation and allocates equal time to generating tokens, irrespective of their importance. These inefficiencies collectively contribute to the suboptimal performance of speculative decoding. To further improve LLM inference, we introduce Cascade Speculative Drafting (CS Drafting), a speculative execution algorithm that incorporates two types of cascades. The Vertical Cascade eliminates autoregressive generation from neural models, while the Horizontal Cascade optimizes time allocation in drafting for improved efficiency. Combining both cascades, CS Drafting achieves greater speedup compared to the baselines in our experiments, while preserving the same output distribution as the target model.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 28 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

21 of 28 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  2. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  5. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 16

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2020Knowledge Distillation: A Surveycited by 1,603
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  10. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  11. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  12. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
4 more
  1. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 59
  2. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  3. 2022Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich Systemscited by 9
  4. 2023Mixed Precision Post Training Quantization of Neural Networks with Sensitivity Guided Searchcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 34; 2025: 31; 2026: 18.

83 papers.

Sort:

By area:

  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  3. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  4. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  6. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  7. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  8. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  9. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 75
  10. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 67
  11. 2024MambaByte: Token-free Selective State Space Modelcited by 59
  12. 2024Faster Cascades via Speculative Decodingcited by 38
38 more
  1. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 37
  2. 2024Block Verification Accelerates Speculative Decodingcited by 30
  3. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  4. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 29
  5. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  6. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  7. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 22
  8. 2025Efficient Reasoning for LLMs through Speculative Chain-of-Thoughtcited by 22
  9. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 22
  10. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 21
  11. 2024Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Treecited by 20
  12. 2025Towards Optimal Multi-draft Speculative Decodingcited by 19
  13. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
  14. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 16
  15. 2025Accelerating Large Language Model Reasoning via Speculative Searchcited by 16
  16. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Accelerationcited by 15
  17. 2024HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Modelscited by 15
  18. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculationcited by 14
  19. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 13
  20. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  21. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  22. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rolloutscited by 12
  23. 2024Efficient Inference for Large Language Model-based Generative Recommendationcited by 11
  24. 2024Root Defence Strategies: Ensuring Safety of LLM at the Decoding Levelcited by 11
  25. 2025A Survey on Parallel Reasoningcited by 11
  26. 2024Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledgecited by 10
  27. 2024SpecHub: Provable Acceleration to Multi-Draft Speculative Decodingcited by 10
  28. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
  29. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decodingcited by 10
  30. 2024SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decodingcited by 9
  31. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  32. 2025PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interactioncited by 8
  33. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Modelscited by 7
  34. 2024Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillationcited by 7
  35. 2025Fractured Chain-of-Thought Reasoningcited by 7
  36. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  37. 2024Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokenscited by 6
  38. 2024FastDraft: How to Train Your Draftcited by 6

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  2. 2026Faster LLM Inference via Sequential Monte Carlo
  3. 2024Decoding Speculative Decoding
  4. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  5. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  6. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models
  7. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
  8. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement