mlsanityData to 7 Oct 2026

Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency

arXiv 2505.17074 ↗First posted 20 May 2025cs.CL cs.AI cs.LGcited by 2

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding accelerates Large Language Model (LLM) inference by employing a small speculative model (SSM) to generate multiple candidate tokens and verify them using the LLM in parallel. This technique has been widely integrated into LLM inference serving systems. However, inference requests typically exhibit uncertain execution time, which poses a significant challenge of efficiently scheduling requests in these systems. Existing work estimates execution time based solely on predicted output length, which could be inaccurate because execution time depends on both output length and token acceptance rate of verification by the LLM. In this paper, we propose a semi-clairvoyant request scheduling algorithm called Least-Attained/Perceived-Service for Speculative Decoding (LAPS-SD). Given a number of inference requests, LAPS-SD can effectively minimize average inference latency by adaptively scheduling requests according to their features during decoding. When the token acceptance rate is dynamic and execution time is difficult to estimate, LAPS-SD maintains multiple priority queues and allows request execution preemption across different queues. Once the token acceptance rate becomes stable, LAPS-SD can accurately estimate the execution time and schedule requests accordingly. Extensive experiments show that LAPS-SD reduces inference latency by approximately 39% compared to state-of-the-art scheduling methods.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 37 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

28 of 37 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models20 Mar
  2. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism6 Jun
  3. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices4 Jun
  4. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  5. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  6. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  8. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  9. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  11. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  12. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2023Large Language Modelscited by 774
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  9. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  10. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  11. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  12. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
4 more
  1. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  2. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  3. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
  4. 2024PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Servicescited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Servingcited by 0
  2. 2026DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding: Performance or Illusion?
  2. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference
  3. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving
  4. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  5. 2024Decoding Speculative Decoding
  6. 2026Faster LLM Inference via Sequential Monte Carlo
  7. 2025SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
  8. 2025Speeding up Speculative Decoding via Sequential Approximate Verification