mlsanityData to 7 Oct 2026

From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models

arXiv 2511.02248 ↗First posted 4 Nov 2025cs.DC cs.LGcited by 0

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

Abstract (arXiv)

Serving large generative models such as LLMs and multi- modal transformers requires balancing user-facing SLOs (e.g., time-to-first-token, time-between-tokens) with provider goals of efficiency and cost reduction. Existing solutions rely on static provisioning or model-level autoscaling, both of which treat the model as a monolith. This coarse-grained resource management leads to degraded performance or significant resource underutilization due to poor adaptability to dynamic inference traffic that is common online. The root cause of this inefficiency lies in the internal structure of generative models: they are executed as graphs of interconnected operators. Through detailed characterization and systematic analysis, we find that operators are heterogeneous in their compute and memory footprints and exhibit diverse sensitivity to workload and resource factors such as batch size, sequence length, and traffic rate. This heterogeneity suggests that the operator, rather than the entire model, is the right granularity for scaling decisions. We propose an operator-level autoscaling framework, which allocates resources at finer (operator)-granularity, optimizing the scaling, batching, and placement based on individual operator profiles. Evaluated on production-scale traces, our approach preserves SLOs with up to 40% fewer GPUs and 35% less energy, or under fixed resources achieves 1.6x higher throughput with 5% less energy. These results show that the operator, rather than the model, is fundamentally a more effective unit for scaling large generative workloads.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 21 of its 62 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

21 of 62 references.

References · 21

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  4. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  5. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 52
  6. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 49
  7. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  8. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
  9. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 23
  10. 2025Serving Large Language Models on Huawei CloudMatrix384cited by 22
  11. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Servingcited by 18
  12. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
9 more
  1. 2025Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Servingcited by 13
  2. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 12
  3. 2025AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructurecited by 12
  4. 2025Hierarchical Autoscaling for Large Language Model Serving with Chironcited by 9
  5. 2025HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Servingcited by 9
  6. 2024ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Servingcited by 6
  7. 2025Towards Efficient and Practical GPU Multitasking in the Era of LLMcited by 6
  8. 2025DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Servingcited by 5
  9. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
  2. 2025Theoretical Foundations of Scaling Law in Familial Models
  3. 2026Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling
  4. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Serving
  5. 2025Taming the Titans: A Survey of Efficient LLM Inference Serving
  6. 2023Splitwise: Efficient generative LLM inference using phase splitting
  7. 2026Performance Optimization and Comparative Analysis of Generative AI Models on Advanced Accelerators
  8. 2024Observational Scaling Laws and the Predictability of Language Model Performance