mlsanityData to 7 Oct 2026

StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving

arXiv 2604.09562 ↗First posted 11 Feb 2026cs.DC cs.AIcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Efficient LLM serving must balance throughput and latency across diverse, bursty workloads. We introduce StreamServe, a disaggregated prefill decode serving architecture that combines metric aware routing across compute lanes with adaptive speculative decoding that tunes speculation depth online from runtime signals. StreamServe comprises four components: StreamScheduler for request orchestration, FlowGuard for multi signal routing, PipeServe Engine for disaggregated prefill decode execution on multi GPU, and SpecuStream for runtime adaptive speculation. We evaluate StreamServe on four benchmarks ALPACA, GSM8K, HUMANEVAL, and SUM with 80 queries each and 320 total using 4 A800 40GB GPUs configured as two stream pairs. Across these workloads, StreamServe reduces latency by 11 to 18 times relative to tensor parallel vLLM baselines and reaches throughput up to 2235 tokens per second on summarization tasks. Time per output token remains stable across configurations, indicating that the gains arise from architectural efficiency rather than token quality degradation. Although evaluated on a single node 4 GPU setup, these results suggest that jointly adapting routing and speculation within a disaggregated framework creates a distinct operating regime for LLM inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 13 of its 26 references that are in the map. First posted on arXiv 11 Feb 2026.

Builds on

13 of 26 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice1 Aug
  2. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 9

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  4. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  5. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  6. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  7. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  8. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  9. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decodingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  2. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
  3. 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing
  4. 2026RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation
  5. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
  6. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
  7. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  8. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices