# Enabling Spatially Fine-Grained DVFS in Neural Processing Units for Energy-Efficient LLM Serving

arXiv: https://arxiv.org/abs/2607.16473 · HTML: https://mlsanity.com/paper/2607.16473/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 18 of its 70 references that are in the map.
- First posted on arXiv 17 Jul 2026.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 18 of 18)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms](https://mlsanity.com/paper/2501.02600/) (2025)
- [SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving](https://mlsanity.com/paper/2408.05235/) (2024)
- [EcoServe: Designing Carbon-Aware AI Inference Systems](https://mlsanity.com/paper/2502.05043/) (2025)
- [Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts](https://mlsanity.com/paper/2503.05066/) (2025)
- [POLCA: Power Oversubscription in LLM Cloud Providers](https://mlsanity.com/paper/2308.12908/) (2023)
- [DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS](https://mlsanity.com/paper/2602.18755/) (2026)
- [Predict; Do not React for Enabling Efficient Fine Grain DVFS in GPUs](https://mlsanity.com/paper/2205.00121/) (2022)
- [AI Greenferencing: Routing AI Inferencing to Green Modular Data Centers with Heron](https://mlsanity.com/paper/2505.09989/) (2025)
- [ReGate: Enabling Power Gating in Neural Processing Units](https://mlsanity.com/paper/2508.02536/) (2025)
- [Analysis of Edge Mismatch and Output Power Degradation in Cascoded Class-D Power Amplifiers Using Dual-Range Voltage Level Shifters](https://mlsanity.com/paper/2602.09820/) (2026)

## Abstract (arXiv)

As neural processing units (NPUs) evolve rapidly to accommodate the ever-increasing compute demand of large language models (LLMs), their power consumption is becoming a limiting factor. Our study shows that using dynamic voltage and frequency scaling (DVFS) to exploit the service-level objective (SLO) slacks is a promising way to improve NPU energy efficiency for LLM services. And as tensor operators in LLMs exhibit diverse bottlenecks across NPU components, it is desirable to configure the frequency separately for each component to maximize their energy efficiency. In this paper, we develop eNPU that enables hardware and software support for spatially fine-grained, component-level DVFS on NPUs. eNPU refactors the NPU core pipeline to partition components into separate V/f domains. It introduces lightweight cross-domain communication mechanisms to mitigate synchronization overheads across components, and extends the NPU ISA for sub-μs DVFS control. eNPU uses a compiler-driven two-level greedy search to co-optimize instruction scheduling and per-component V/f selection under SLO constraints. We implement eNPU's pipeline design on an open-source NPU core to verify its functionality and evaluate the energy savings with a production-level NPU simulator with various LLMs using production traces. eNPU reduces energy consumption of LLM services by 25.8%--35.2% with 3.45% area overhead on a TPUv4 chip, while preserving strict SLO guarantees.