Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
To meet the ever-increasing computing demands of large language model (LLM) services, modern cloud platforms have widely deployed neural processing units (NPUs). These NPU chips have been developed and evolved at an incredibly fast pace, this inevitably produces heterogeneous compute pools backed by different versions of NPU chips. Unfortunately, due to the lack of system and architecture support for managing NPU heterogeneity in the cloud, it is unclear how to best utilize heterogeneous NPUs to maximize the energy and cost efficiency for LLM services. In this paper, we first conduct a characterization study of various generations of real NPU chips to demonstrate the potential benefits on energy/cost efficiency and performance by utilizing heterogeneous NPU chips. To realize these benefits, we present NeuScale, an auto-scaling framework to automatically exploit heterogeneous NPUs for cloud platforms. NeuScale manages heterogeneous NPU resources with a new vPod abstraction, which abstracts the core hardware parameters of different NPU versions and provides compatibility with existing ML frameworks. It makes the best-fit vPod allocations for different LLM inference requests using an intuitive and lightweight roofline-based analysis. It supports fine-grained dynamic NPU resource provisioning by adjusting both the vPod configuration (i.e., scaling up/down) and the number of vPods (e.g., scaling in/out). To validate the benefits of NeuScale at scale, we implement it with a production-level NPU simulator. Our evaluation with popular LLMs shows that NeuScale can significantly improve cost efficiency and service-level objective (SLO) satisfaction rate by best utilizing heterogeneous NPU resources.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 88 references that are in the map. First posted on arXiv 17 Jul 2026.
Builds on
31 of 88 references.
Closest ancestors · 4
Same direction, newest first
- 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
- 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
- 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan
- 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
Other references · 27
- 2017Attention Is All You Needcited by 59,932
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
- 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
- 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
- 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
- 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
- 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
15 more
- 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
- 2024Large Multimodal Agents: A Surveycited by 99
- 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
- 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
- 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
- 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
- 2020FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented Microservicescited by 41
- 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
- 2024BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Cachingcited by 23
- 2025Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Expertscited by 14
- 2025Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusterscited by 12
- 2025Life-Cycle Emissions of AI Hardware: A Cradle-To-Grave Approach and Generational Trendscited by 10
- 2025Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelismcited by 9
- 2024Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMscited by 4
- 2025HAS-GPU: Efficient Hybrid Auto-scaling with Fine-grained GPU Allocation for SLO-aware Serverless Inferencescited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Enabling Spatially Fine-Grained DVFS in Neural Processing Units for Energy-Efficient LLM Serving
- 2026NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement
- 2026AscendKernelGen: A Systematic Study of LLM-Based Kernel Generation for Neural Processing Units
- 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Serving
- 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
- 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference
- 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments
- 2026AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall ↗