mlsanityData to 7 Oct 2026

AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding

arXiv 2501.12162 ↗First posted 21 Jan 2025cs.CL cs.AI cs.DCcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Modern large language model (LLM) applications exhibit diverse service-level objectives (SLOs), from low-latency requirements in interactive coding assistants to more relaxed constraints in data wrangling tasks. Existing LLM serving systems, which rely on uniform batching and scheduling strategies, often fail to meet these heterogeneous SLOs concurrently. We present AdaServe, the first LLM serving system designed to support efficient multi-SLO serving through SLO-customized speculative decoding. AdaServe formulates multi-SLO serving as a constrained optimization problem and introduces a hardware-aware algorithm that constructs a speculation tree tailored to each request's latency target. It features a speculate-select-verify pipeline that enables fine-grained control over decoding speed while maximizing system throughput. AdaServe further adapts to workload variation by dynamically adjusting speculation parameters. Evaluations across diverse workloads show that AdaServe reduces SLO violations by up to 4.3× and improves goodput by up to 1.9× compared to the best performing baselines, highlighting its effectiveness in multi-SLO serving.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 55 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

27 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  2. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  3. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  6. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  11. 2019MLPerf Inference Benchmarkcited by 195
  12. 2022Can Foundation Models Wrangle Your Data?cited by 173
6 more
  1. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  2. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  3. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  4. 2023Fairness in Serving Large Language Modelscited by 77
  5. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  6. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Draftercited by 12

Similar, not cited

Close in wording, no citation either way

  1. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
  2. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMs
  3. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving
  4. 2023Accelerating LLM Inference with Staged Speculative Decoding
  5. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  6. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding
  7. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  8. 2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications