mlsanityData to 7 Oct 2026

FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving

arXiv 2604.20503 ↗First posted 22 Apr 2026cs.DCcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD) is a widely used approach for accelerating decode-heavy LLM inference workloads. While online inference workloads are highly dynamic, existing SD systems are rigid and take a coarse-grained approach to SD management. They typically set the speculative token length for an entire batch and serialize the execution of the draft and verification phases. Consequently, these systems fall short at adapting to volatile online inference traffic. Under low load, they exhibit prolonged latency because the draft phase blocks the verification phase for the entire batch, leaving GPU computing resources underutilized. Conversely, under high load, they waste computation on rejected tokens during the verification phase, overloading GPU resources. We introduce FASER, a novel system that features fine-grained SD phase management. First, FASER minimizes computational waste by dynamically adjusting the speculative length for each request within a continuous batch and by performing early pruning of rejected tokens inside the verification phase. Second, FASER breaks the verification phase into frontiers, or chunks, to overlap them with the draft phase. This overlap is achieved via fine-grained spatial multiplexing with minimal resource interference. Our FASER prototype in vLLM improves throughput by up to 53% and reduces latency by up to 1.92× compared to state-of-the-art systems.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 54 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

33 of 54 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving27 Dec
  2. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithms21 May
  3. 2025SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism16 May
  4. 2025PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding2 May
  5. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting11 Apr
  6. 2025SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding5 Apr
  7. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving7 Mar
  8. 2025TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding21 Feb
  9. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation27 Nov
  10. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  11. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decoding8 Oct
  12. 2024Mixture of Attentions For Speculative Decoding4 Oct
9 more
  1. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  2. 2024PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation16 Jul
  3. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices4 Jun
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  6. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding9 Oct
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 12

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  9. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  10. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  11. 2024Forecasting GPU Performance for Deep Learning Training and Inferencecited by 15
  12. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026CAST: Cost-Aware Speculative Trees from One-Pass Block Drafterscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  2. 2026Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices
  3. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference
  4. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding
  5. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  6. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
  7. 2025SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
  8. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving