mlsanityData to 7 Oct 2026

OpScale: Operator-level Provisioning and Autoscaling for LLM Serving

arXiv 2608.13499 ↗First posted 13 Aug 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Achieving cost efficiency while meeting strict user-facing SLOs (e.g., time-to-first-token) remains a fundamental challenge for cloud GPU clusters serving large language models (LLMs). Autoscaling is the key mechanism for cluster resource management, yet a basic system design question is open for serving LLMs: what should be the unit of scaling? Existing approaches primarily treat the entire model as a monolithic scaling unit--simple but unable to capture the fine-grained dynamics of inference workloads. As a result, such coarse-grained scaling often leads to either SLO violations under bursty demand or significant GPU under-utilization. Our characterization reveals substantial operator heterogeneity, exposing operator-level elasticity as a viable scaling primitive. We present OpScale, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving. OpScale is designed to tackle the high complexity and the space explosion problem, arising from operating at this finer granularity. Evaluated with production traces on up to 40 A100s and 24 GB200s, OpScale attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 55 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

26 of 55 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage28 Apr
  2. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  4. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 20

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  4. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  5. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  6. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 52
  7. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  8. 2020FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented Microservicescited by 41
  9. 2025Measuring Agents in Productioncited by 35
  10. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
  11. 2025Serving Large Language Models on Huawei CloudMatrix384cited by 22
  12. 2025DeepServe: Serverless Large Language Model Serving at Scalecited by 17
8 more
  1. 2025Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Servingcited by 13
  2. 2022Autothrottle: A Practical Bi-Level Approach to Resource Management for SLO-Targeted Microservicescited by 12
  3. 2025AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructurecited by 12
  4. 2025Hierarchical Autoscaling for Large Language Model Serving with Chironcited by 9
  5. 2025LithOS: An Operating System for Efficient Machine Learning on GPUscited by 9
  6. 2025HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Servingcited by 9
  7. 2025Towards Efficient and Practical GPU Multitasking in the Era of LLMcited by 6
  8. 2025DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Servingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  2. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments
  3. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  4. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  5. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
  6. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  7. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  8. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems