mlsanityData to 14 Oct 2026

TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput

arXiv 2406.14066 ↗First posted 20 Jun 2024cs.AI cs.PFcited by 37

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large Language Model (LLM) serving systems batch concurrent user requests to achieve efficient serving. However, in real-world deployments, such inter-request parallelism from batching is often limited by external factors such as low request rates or memory constraints. Recent works focus on intra-request parallelism from speculative decoding as a solution to this problem. Unfortunately, benefits from intra-request parallelism are often fragile, as speculative decoding causes overhead, and speculated tokens may miss. We observe that speculative decoding may degrade LLM serving performance if added naively without tuning to the incoming requests and the speculation method. To alleviate the need for expert tuning and make speculative decoding more robust, we present TurboSpec, a speculation control system that automatically profiles the execution environment and utilizes a feedback-based algorithm to dynamically adjust the amount of intra-request parallelism in LLM serving. TurboSpec predicts "goodput" - the amount of successfully generated tokens - to evaluate and adjust intra-request parallelism amount to that with the highest goodput in runtime. We implement TurboSpec on a real-world LLM serving system vLLM and demonstrate its effectiveness across diverse workloads and hardware configurations, providing consistent performance improvements across all test scenarios.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 38 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

18 of 38 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  3. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models28 Oct
  6. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  7. 2023Online Speculative Decoding11 Oct
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 9

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2022Efficiently Scaling Transformer Inferencecited by 539
  8. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 73
  9. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 15; 2026: 16.

37 papers.

Sort:

By area:

  1. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 91
  2. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 36
  3. 2024Dynamic Depth Decoding: Faster Speculative Decoding for LLMscited by 20
  4. 2025TETRIS: Optimal Draft Token Selection for Batch Speculative Decodingcited by 15
  5. 2025Scaling Speculative Decoding with Lookahead Reasoningcited by 13
  6. 2025Utility-Driven Speculative Decoding for Mixture-of-Expertscited by 13
  7. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarioscited by 12
  8. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  9. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Modelscited by 9
  10. 2025SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decodingcited by 9
  11. 2025What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decodingcited by 8
  12. 2025Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Designcited by 8
25 more
  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decodingcited by 7
  2. 2025Collaborative Speculative Inference for Efficient LLM Inference Servingcited by 5
  3. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 5
  4. 2025TokenSwift: Lossless Acceleration of Ultra Long Sequence Generationcited by 4
  5. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  6. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMscited by 4
  7. 2025Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problemcited by 3
  8. 2024SSSD: Simply-Scalable Speculative Decodingcited by 2
  9. 2025HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decodingcited by 2
  10. 2024Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernelscited by 1
  11. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  12. 2026MineDraft: A Framework for Batch Parallel Speculative Decodingcited by 1
  13. 2026Optimus: Elastic Decoding for Efficient Diffusion LLM Servingcited by 1
  14. 2026Enabling KV Caching of Shared Prefix for Diffusion Language Modelscited by 1
  15. 2026Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Offcited by 1
  16. 2026TreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative Decodingcited by 1
  17. 2025OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputscited by 0
  18. 2026Regulating Branch Parallelism in LLM Servingcited by 0
  19. 2026An Interpretable Latency Model for Speculative Decoding in LLM Servingcited by 0
  20. 2026Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detectioncited by 0
  21. 2026DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMscited by 0
  22. 2026OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Servingcited by 0
  23. 2026DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Servingcited by 0
  24. 2026AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fillcited by 0
  25. 2026DLoop: Looped Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
  2. 2026SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
  3. 2025SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding
  4. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
  5. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  6. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  7. 2026WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve