# WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

arXiv: https://arxiv.org/abs/2607.02391 · HTML: https://mlsanity.com/paper/2607.02391/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 14 of 14)

- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization](https://mlsanity.com/paper/2505.06371/) (2025)
- [Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems](https://mlsanity.com/paper/2407.04014/) (2024)
- [TokenPowerBench: Benchmarking the Power Consumption of LLM Inference](https://mlsanity.com/paper/2512.03024/) (2025)
- [SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving](https://mlsanity.com/paper/2408.05235/) (2024)
- [LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferences](https://mlsanity.com/paper/2410.02950/) (2024)
- [MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI](https://mlsanity.com/paper/2410.12032/) (2024)
- [Quantifying the Energy Consumption and Carbon Emissions of LLM Inference via Simulations](https://mlsanity.com/paper/2507.11417/) (2025)
- [From Prompts to Power: Measuring the Energy Footprint of LLM Inference](https://mlsanity.com/paper/2511.05597/) (2025)
- [Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling](https://mlsanity.com/paper/2508.00904/) (2025)
- [Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures](https://mlsanity.com/paper/2604.09048/) (2026)
- [This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMs](https://mlsanity.com/paper/2603.15699/) (2026)
- [The Energy Cost of Execution-Idle in GPU Clusters](https://mlsanity.com/paper/2604.04745/) (2026)

## Built on by (most cited first, 1 of 1)

- [The Right Information Extraction Pipeline Depends on the Document: Accuracy-Energy Trade-offs for Small, Local Models](https://mlsanity.com/paper/2609.31341/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination. While some predictive models exist, they still require profiling data and struggle to generalize to hardware unseen during training. To address this, we introduce textitWattGPU, featuring two predictive models for mean GPU power draw and Inter-Token Latency (ITL). Our approach leverages only publicly available LLM metadata and GPU specifications, eliminating the need for hardware access or profiling while enabling generalization to unseen NVIDIA server-grade GPUs and LLMs. We evaluate our models using rigorous leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs (0.1B--27B parameters) and 8 GPUs under both offline and server scenarios. The mean power draw model achieves a median absolute percentage error of ≤3.4% for offline and ≤13.5% for server scenarios on unseen GPUs, while the latency model achieves ≤8.5% in server mode, both maintaining strong GPU ranking correlations for server scenarios (Kendall τ≥0.76). Compared to standard physically grounded baselines -- Load-Scaled Thermal Design Power (TDP) for power draw and roofline for latency -- our models reduce median absolute percentage error by approximately 4× on unseen LLM-GPU combinations for server scenarios or approximately 2× for completely unseen GPUs. WattGPU's data and code are publicly available at https://github.com/maufadel/wattgpu.