mlsanityData to 7 Oct 2026

Faster Cascades via Speculative Decoding

arXiv 2405.19261 ↗First posted 29 May 2024cs.CL cs.AI cs.LGcited by 38

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Cascades and speculative decoding are two common approaches to improving language models' inference efficiency. Both approaches involve interleaving models of different sizes, but via fundamentally distinct mechanisms: cascades employ a deferral rule that invokes the larger model only for "hard" inputs, while speculative decoding uses speculative execution to primarily invoke the larger model in parallel verification mode. These mechanisms offer different benefits: empirically, cascades offer better cost-quality trade-offs, often even outperforming the large model, while theoretically, speculative decoding offers a guarantee of quality-neutrality. In this paper, we leverage the best of both these approaches by designing new speculative cascading techniques that implement their deferral rule through speculative execution. We characterize the optimal deferral rule for our speculative cascades, and employ a plug-in approximation to the optimal rule. Experiments with Gemma and T5 models on a range of language benchmarks show that our approach yields better cost quality trade-offs than cascading and speculative decoding baselines.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 57 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

40 of 57 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting29 Apr
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  4. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  5. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  6. 2023PaSS: Parallel Speculative Sampling22 Nov
  7. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  8. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  9. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  11. 2023Speculative Decoding with Big Little Decoder15 Feb
  12. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
1 more
  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2015Teaching Machines to Read and Comprehendcited by 1,039
  11. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
15 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  2. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  3. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  4. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  5. 2022Confident Adaptive Language Modelingcited by 297
  6. 2022UL2: Unifying Language Learning Paradigmscited by 292
  7. 2019Depth-Adaptive Transformercited by 227
  8. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  9. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  10. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
  11. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  12. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  13. 2020Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Modelscited by 39
  14. 2022TangoBERT: Reducing Inference Cost by using Cascaded Architecturecited by 13
  15. 2022BabyBear: Cheap inference triage for expensive language modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 19; 2026: 18.

38 papers.

Sort:

By area:

  1. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  2. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  3. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  4. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 29
  5. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  6. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  7. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  8. 2025AutoJudge: Judge Decoding Without Manual Annotationcited by 13
  9. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
  10. 2025LLM Routing with Dueling Feedbackcited by 6
  11. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  12. 2026DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verificationcited by 5
26 more
  1. 2025Learning to Partially Defer for Sequencescited by 4
  2. 2025TokenSwift: Lossless Acceleration of Ultra Long Sequence Generationcited by 4
  3. 2025Economic Evaluation of LLMscited by 4
  4. 2025SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verificationcited by 4
  5. 2026Overcoming Joint Intractability with Lossless Hierarchical Speculative Decodingcited by 4
  6. 2026Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Samplingcited by 4
  7. 2025textttSPECS: Faster Test-Time Scaling through Speculative Draftscited by 3
  8. 2026When More Experts Hurt: Underfitting in Multi-Expert Learning to Defercited by 2
  9. 2026A Practical Investigation of Training-free Relaxed Speculative Decodingcited by 2
  10. 2026Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Makingcited by 2
  11. 2025From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascadescited by 1
  12. 2026TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verificationcited by 1
  13. 2026Density-Ratio Losses for Post-Hoc Learning to Defercited by 1
  14. 2026Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systemscited by 1
  15. 2025Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rolloutscited by 0
  16. 2025Fast Inference via Hierarchical Speculative Decodingcited by 0
  17. 2025RAGFort: Dual-Path Defense Against Proprietary Knowledge Base Extraction in Retrieval-Augmented Generationcited by 0
  18. 2026Beyond the Target: From Imitation to Collaboration in Speculative Decodingcited by 0
  19. 2026Diffusion Language Model Parallel Decoding via Product-of-Experts Bridgecited by 0
  20. 2026VIA-SD: Verification via Intra-Model Routing for Speculative Decodingcited by 0
  21. 2026Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMscited by 0
  22. 2026On-Demand Attention: Language Models Know When to Recallcited by 0
  23. 2026Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decodingcited by 0
  24. 2026UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decodingcited by 0
  25. 2026Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Headscited by 0
  26. 2026Secure Speculative Decoding for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Disparate Impacts of Speculative Decoding
  2. 2026Speculative Decoding with a Speculative Vocabulary
  3. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  4. 20253-Model Speculative Decoding
  5. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  6. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  7. 2026Speculative Decoding and the Curse of Multilinguality
  8. 2024Decoding Speculative Decoding