mlsanityData to 14 Oct 2026

FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving

arXiv 2610.06917 ↗First posted 2 Oct 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Prefill-decode disaggregation is becoming a common architecture for LLM serving because it separates two phases with distinct execution patterns and SLO objectives. Existing systems typically combine a fixed prefill/decode worker ratio with request routing across workers. However, real-world workloads exhibit both short bursts and sustained shifts in the prefill-to-decode demand ratio. As a result, a configuration that is well provisioned at one time may quickly become mismatched, causing latency SLO violations even when idle capacity exists elsewhere. Existing autoscaling mechanisms can add capacity, but they react slowly, require spare GPUs, and do not directly address short-timescale phase imbalance. We present FluidPD, a P/D-disaggregated serving system that provides SLO-aware in-place elasticity. FluidPD introduces two complementary mechanisms. FluidToken handles transient imbalance by offloading a bounded portion of prefill computation to decode workers when decode-side slack is available. FluidRole handles sustained imbalance by reassigning running workers between prefill and decode roles in place, avoiding model reload and engine restart. Both mechanisms are guided by lightweight pressure indices that expose prefill and decode-side resource pressure before they appear as SLO violations. Across production Azure trace workloads, FluidPD improves overall SLO attainment over static SGLang by up to 94.6 percentage points, demonstrating that SLO-aware in-place P/D elasticity improves service quality without provisioning additional workers.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 53 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

35 of 53 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving2 Jul
  2. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity3 Dec
  3. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving4 Aug
  4. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice1 Aug
  5. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference29 May
  6. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture17 May
  7. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb
  8. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  9. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  10. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  11. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  12. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
1 more
  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 22

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  8. 2022Efficiently Scaling Transformer Inferencecited by 539
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  11. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  12. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
10 more
  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  2. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  3. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  4. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  5. 2024Pie: Pooling CPU Memory for LLM Inferencecited by 13
  6. 2025AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructurecited by 13
  7. 2025Beyond the Buzz: A Pragmatic Take on Inference Disaggregationcited by 9
  8. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Servingcited by 6
  9. 2026Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Servingcited by 6
  10. 2025Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inferencecited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
  2. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  3. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving
  4. 2026DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS
  5. 2026LAPS: A Length-Aware-Prefill LLM Serving System
  6. 2026Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexing
  7. 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure
  8. 2025OOCO: Latency-disaggregated Architecture for Online-Offline Co-locate LLM Serving