mlsanityData to 14 Oct 2026

Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference

arXiv 2508.19559 ↗First posted 27 Aug 2025cs.DC cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Serving Large Language Models (LLMs) is a GPU-intensive task where traditional autoscalers fall short, particularly for modern Prefill-Decode (P/D) disaggregated architectures. This architectural shift, while powerful, introduces significant operational challenges, including inefficient use of heterogeneous hardware, network bottlenecks, and critical imbalances between prefill and decode stages. We introduce HeteroScale, a coordinated autoscaling framework that addresses the core challenges of P/D disaggregated serving. HeteroScale combines a topology-aware scheduler that adapts to heterogeneous hardware and network constraints with a novel metric-driven policy derived from the first large-scale empirical study of autoscaling signals in production. By leveraging a single, robust metric to jointly scale prefill and decode pools, HeteroScale maintains architectural balance while ensuring efficient, adaptive resource management. Deployed in a massive production environment on tens of thousands of GPUs, HeteroScale has proven its effectiveness, increasing average GPU utilization by a significant 26.6 percentage points and saving hundreds of thousands of GPU-hours daily, all while upholding stringent service level objectives.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 23 of its 52 references that are in the map. First posted on arXiv 27 Aug 2025.

Builds on

23 of 52 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  11. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  12. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
11 more
  1. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  2. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  3. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  4. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  5. 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43
  6. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
  7. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  8. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  9. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 31
  10. 2025SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inferencecited by 10
  11. 2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025UCCL-EP: Portable Expert-Parallel Communicationcited by 12
  2. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  3. 2026Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inferencecited by 0
  4. 2026Cross-Model Autoscaling for Shared LLM Servingcited by 0
  5. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Servingcited by 0