mlsanityData to 7 Oct 2026

AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving

arXiv 2503.05096 ↗First posted 7 Mar 2025cs.CLcited by 18

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Cloud-based Large Language Model (LLM) services often face challenges in achieving low inference latency and meeting Service Level Objectives (SLOs) under dynamic request patterns. Speculative decoding, which exploits lightweight models for drafting and LLMs for verification, has emerged as a compelling technique to accelerate LLM inference. However, existing speculative decoding solutions often fail to adapt to fluctuating workloads and dynamic system environments, resulting in impaired performance and SLO violations. In this paper, we introduce AdaSpec, an efficient LLM inference system that dynamically adjusts speculative strategies according to real-time request loads and system configurations. AdaSpec proposes a theoretical model to analyze and predict the efficiency of speculative strategies across diverse scenarios. Additionally, it implements intelligent drafting and verification algorithms to maximize performance while ensuring high SLO attainment. Experimental results on real-world LLM service traces demonstrate that AdaSpec consistently meets SLOs and achieves substantial performance improvements, delivering up to 66% speedup compared to state-of-the-art speculative inference systems. The source code is publicly available at https://github.com/cerebellumking/AdaSpec

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 44 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

24 of 44 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  2. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  3. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  7. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  8. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  9. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  10. 2023Speculative Decoding with Big Little Decoder15 Feb
  11. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  12. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  9. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  10. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  11. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  12. 2024Queue management for slo-oriented large language model servingcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 13.

18 papers.

Sort:

By area:

  1. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Servingcited by 13
  2. 2025Utility-Driven Speculative Decoding for Mixture-of-Expertscited by 12
  3. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarioscited by 11
  4. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Trainingcited by 8
  5. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  6. 2025SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Modelscited by 2
  7. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  8. 2026MineDraft: A Framework for Batch Parallel Speculative Decodingcited by 1
  9. 2026FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Servingcited by 1
  10. 2026Adversarial Prompts for Acceptance Collapse in Speculative Decodingcited by 1
  11. 2026Regulating Branch Parallelism in LLM Servingcited by 0
  12. 2026Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decodingcited by 0
6 more
  1. 2026CONCORD: Asynchronous Sparse Aggregation for Device-Cloud RAG under Document Isolationcited by 0
  2. 2026Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detectioncited by 0
  3. 2026AgentSpec: Speculative Decoding for Batch Inference of LLM Agentscited by 0
  4. 2026OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Servingcited by 0
  5. 2026TreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative Decodingcited by 0
  6. 2026Crossflow: Prefill-Decode Elasticity for Agentic LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
  2. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  3. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
  4. 2026SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
  5. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving
  6. 2026WANSpec: Leveraging Global Compute Capacity for LLM Inference
  7. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  8. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices