mlsanityData to 7 Oct 2026

MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding

arXiv 2408.11049 ↗First posted 20 Aug 2024cs.CLcited by 87

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large Language Models (LLMs) have become more prevalent in long-context applications such as interactive chatbots, document analysis, and agent workflows, but it is challenging to serve long-context requests with low latency and high throughput. Speculative decoding (SD) is a widely used technique to reduce latency losslessly, but the conventional wisdom suggests that its efficacy is limited to small batch sizes. In MagicDec, we show that surprisingly SD can achieve speedup even for a high throughput inference regime for moderate to long sequences. More interestingly, an intelligent drafting strategy can achieve better speedup with increasing batch size based on our rigorous analysis. MagicDec first identifies the bottleneck shifts with increasing batch size and sequence length, and uses these insights to deploy SD more effectively for high throughput inference. We leverage draft model with sparse KV cache to address the KV bottleneck, which scales with both sequence length and batch size. Additionally, we propose a theoretical model to select the optimal drafting strategy for maximum speedup. Our work highlights the broad applicability of speculative decoding in long-context serving, as it can enhance throughput and reduce latency without compromising accuracy. For moderate to long sequences, we demonstrate up to 2.51x speedup for Llama3.1-8B when serving batch sizes ranging from 32 to 256 on various types of hardware and tasks.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 52 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

28 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput20 Jun
  2. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  3. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models28 Oct
  4. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  5. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  6. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
10 more
  1. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  4. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  5. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  6. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  7. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  8. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 70
  9. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  10. 2021SpreadsheetCoder: Formula Prediction from Semi-structured Contextcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 45; 2026: 37.

87 papers.

Sort:

By area:

  1. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 54
  4. 2025LongCat-Flash Technical Reportcited by 54
  5. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 29
  6. 2025Cognitive Memory in Large Language Modelscited by 28
  7. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  8. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  9. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  10. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabulariescited by 20
  11. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  12. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 19
38 more
  1. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 17
  2. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  3. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  4. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  5. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarioscited by 11
  6. 2025Speculative Decoding Reimagined for Multimodal Large Language Modelscited by 10
  7. 2025Rectified Sparse Attentioncited by 10
  8. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  9. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  10. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 9
  11. 2024Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillationcited by 7
  12. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decodingcited by 7
  13. 2025R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoningcited by 7
  14. 2025Efficient Speculative Decoding for Llama at Scale: Challenges and Solutionscited by 7
  15. 2025SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verificationcited by 7
  16. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decodingcited by 7
  17. 2025Draft-based Approximate Inference for LLMscited by 6
  18. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Servingcited by 6
  19. 2026P-EAGLE: Parallel-Drafting EAGLE with Scalable Trainingcited by 5
  20. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 5
  21. 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMscited by 5
  22. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  23. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 5
  24. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 5
  25. 2025TokenSwift: Lossless Acceleration of Ultra Long Sequence Generationcited by 4
  26. 2025SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequencescited by 4
  27. 2025OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decodingcited by 4
  28. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerializationcited by 4
  29. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  30. 2025DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Modelscited by 4
  31. 2025CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMscited by 4
  32. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decodingcited by 4
  33. 2026KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problemcited by 4
  34. 2026Attention Drift: What Autoregressive Speculative Decoding Models Learncited by 4
  35. 2026D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decodingcited by 4
  36. 2025Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMscited by 3
  37. 2025Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Designcited by 3
  38. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMscited by 3

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Decoding Speculative Decoding
  2. 2025Speculative Decoding: Performance or Illusion?
  3. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving
  4. 2025OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs
  5. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  7. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  8. 2023Accelerating LLM Inference with Staged Speculative Decoding