mlsanityData to 14 Oct 2026

Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture

arXiv 2505.11916 ↗First posted 17 May 2025cs.DCcited by 5

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Existing large language model (LLM) serving systems typically employ Prefill-Decode disaggregated architecture to prevent computational interference between the prefill and decode phases. However, in real-world LLM serving scenarios, significant fluctuations in request input/output lengths lead to imbalanced computational loads between prefill and decode nodes under traditional static node allocation strategies, consequently preventing efficient utilization of computing resources to improve the system's goodput. To address this challenge, we design and implement Arrow, an adaptive scheduler that leverages stateless instances and latency characteristics of prefill and decode tasks to achieve efficient adaptive request and instance scheduling. Arrow dynamically adjusts the number of instances handling prefill and decode tasks based on real-time cluster performance metrics, substantially enhancing the system's capability to handle traffic spikes and load variations. Our evaluation under diverse real-world workloads shows that Arrow achieves up to 2.55 × higher request serving rates compared to state-of-the-art Prefill-Decode disaggregated serving systems.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 54 references that are in the map. First posted on arXiv 17 May 2025.

Builds on

35 of 54 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage28 Apr
  2. 2025Taming the Titans: A Survey of Efficient LLM Inference Serving28 Apr
  3. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving12 Apr
  4. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation26 Mar
  5. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  6. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
  7. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  8. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  9. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
  10. 2024A Survey on Efficient Inference for Large Language Models22 Apr
  11. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
5 more
  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insights26 Feb
  2. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  4. 2023Fairness in Serving Large Language Models31 Dec
  5. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  12. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
6 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  2. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  3. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  4. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  5. 2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20
  6. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scalingcited by 3
  2. 2025Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUscited by 3
  3. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  4. 2026Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inferencecited by 0
  5. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
  2. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  3. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  4. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  5. 2025VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing
  6. 2024Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing
  7. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  8. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving