mlsanityData to 7 Oct 2026

Speculative Decoding: Performance or Illusion?

arXiv 2601.11580 ↗First posted 31 Dec 2025cs.CL cs.AIcited by 13

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD) has become a popular technique to accelerate Large Language Model (LLM) inference, yet its real-world effectiveness remains unclear as prior evaluations rely on research prototypes and unrealistically small batch sizes. We present, to our knowledge, the first systematic study of SD on a production-grade and widely deployed inference engine (vLLM), covering multiple SD variants (n-gram, EAGLE/EAGLE-3, Draft-Model, Multi-Token Prediction) across diverse workloads, model scales, and batch sizes. We analyze key factors governing SD performance, and quantify a theoretical upper bound on SD speedup. Our results show that verification by the target model dominates the execution, while acceptance length varies markedly across output token positions, requests, and datasets. Comparing measured performance with theoretical bounds reveals substantial gaps between observed and theoretical upper bounds, and we leverage this observation to highlight new research opportunities that our study opens up in improving SD.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 37 references that are in the map. First posted on arXiv 31 Dec 2025.

Builds on

23 of 37 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2024PLD+: Accelerating LLM inference by leveraging Language Model Artifacts2 Dec
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  4. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  5. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  7. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  8. 2023Online Speculative Decoding11 Oct
  9. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  10. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2015Teaching Machines to Read and Comprehendcited by 1,039
  10. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  11. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  12. 2023InstructCoder: Instruction Tuning Large Language Models for Code Editingcited by 28
1 more
  1. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026LK Losses: Direct Acceptance Rate Optimization for Speculative Decodingcited by 17
  2. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarioscited by 11
  3. 2026Multi-Token Prediction via Self-Distillationcited by 6
  4. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 5
  5. 2026D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decodingcited by 4
  6. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decodingcited by 4
  7. 2026Bole: Efficient Tree Speculation for Hybrid-Attention Language Modelscited by 3
  8. 2026A Practical Investigation of Training-free Relaxed Speculative Decodingcited by 2
  9. 2026K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modelingcited by 1
  10. 2026Adversarial Prompts for Acceptance Collapse in Speculative Decodingcited by 1
  11. 2026Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decodingcited by 0
  12. 2026TreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative Decodingcited by 0
1 more
  1. 2026AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelinescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  2. 2024Decoding Speculative Decoding
  3. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  4. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  5. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  6. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference
  7. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  8. 2023Accelerating LLM Inference with Staged Speculative Decoding