mlsanityData to 7 Oct 2026

The Synergy of Speculative Decoding and Batching in Serving Large Language Models

arXiv 2310.18813 ↗First posted 28 Oct 2023cs.LG cs.DCcited by 19

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large Language Models (LLMs) like GPT are state-of-the-art text generation models that provide significant assistance in daily routines. However, LLM execution is inherently sequential, since they only produce one token at a time, thus incurring low hardware utilization on modern GPUs. Batching and speculative decoding are two techniques to improve GPU hardware utilization in LLM inference. To study their synergy, we implement a prototype implementation and perform an extensive characterization analysis on various LLM models and GPU architectures. We observe that the optimal speculation length depends on the batch size used. We analyze the key observation and build a quantitative model to explain it. Based on our analysis, we propose a new adaptive speculative decoding strategy that chooses the optimal speculation length for different batch sizes. Our evaluations show that our proposed method can achieve equal or better performance than the state-of-the-art speculation decoding schemes with fixed speculation length.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 18 references that are in the map. First posted on arXiv 28 Oct 2023.

Builds on

12 of 18 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Online Speculative Decoding11 Oct
  2. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  6. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  7. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  8. 2018PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systemscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 8; 2026: 3.

19 papers.

Sort:

By area:

  1. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  2. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  3. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodputcited by 36
  4. 2024Decoding Speculative Decodingcited by 33
  5. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  6. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 19
  7. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 17
  8. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  9. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  10. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Modelscited by 9
  11. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Modelscited by 7
  12. 2025Efficient Speculative Decoding for Llama at Scale: Challenges and Solutionscited by 7
7 more
  1. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Servingcited by 6
  2. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Modelscited by 5
  3. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 4
  4. 2024SSSD: Simply-Scalable Speculative Decodingcited by 2
  5. 2025SpecMemo: Speculative Decoding is in Your Pocketcited by 1
  6. 2026Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edgecited by 0
  7. 2026DLoop: Looped Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
  2. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
  3. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting
  4. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
  5. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts
  6. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  7. 2026MineDraft: A Framework for Batch Parallel Speculative Decoding
  8. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference