mlsanityData to 14 Oct 2026

SARA: SLO-Aware Resource Allocation for Disaggregated Agentic LLM Services

arXiv 2609.26763 ↗First posted 22 Sep 2026cs.PF cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Recent advances in large language models (LLMs) are driving the emergence of multi-modal and agentic services for mobile users through cloud and edge infrastructures, where long-context workloads pose daunting challenges for inference latency. Existing disaggregated LLM serving systems largely rely on hardware profiling, configuration enumeration, or heuristic scheduling, offering limited analytical guidance for cost-efficient resource allocation. In this paper, we propose SARA, a Service level objectives (SLOs)-Aware Resource Allocation framework for disaggregated agentic LLM serving systems, which maximizes goodput under a deployment cost constraint and a series of quantile-based SLO constraints. By capitalizing on queuing theory, we first model the prefill, KV cache transfer, and decode stages as an M/G/k queue, an M/G/1 queue, and a generalized birth-death process, respectively. The analysis reveals that the prefill and decode stages are dominantly limited by computational capacity and high-bandwidth memory (HBM) resources, respectively. With these mathematical models, we further derive tractable tail behaviors of different stage-wise service level metrics for both light- and heavy-tailed workloads. These characterizations explicitly map workload, model architecture, and hardware parameters to stage-wise SLO constraints and minimum resource requirements. Finally, we develop an effective resource allocation framework to maximize system goodput under limited cost budgets. Simulation and hardware results demonstrate that the proposed framework accurately predicts the stage-wise SLO with mean errors below 5%, and improves system goodput by 26.6% on average over state-of-the-art baseline methods under the same deployment cost.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 43 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

28 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production15 May
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  12. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
12 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  6. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  7. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  8. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  9. 2025Token Communications: A Large Model-Driven Framework for Cross-modal Context-aware Semantic Communicationscited by 35
  10. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Servingcited by 18
  11. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacentercited by 11
  12. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  2. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  3. 2026End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services
  4. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
  5. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information
  6. 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference
  7. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
  8. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving