mlsanityData to 7 Oct 2026

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

arXiv 2604.09048 ↗First posted 10 Apr 2026cs.DC cs.AIcited by 5

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

While the large energy consumption of Large Language Models (LLMs) is recognized by the community, system operators lack guidance for energy-efficient LLM inference deployments that leverage energy trade-offs of heterogeneous hardware due to a lack of energy-aware benchmarks and data. In this work we address this gap with Watt Counts: the largest open-access dataset of energy consumption of LLMs, with over 5,000 experiments for 50 LLMs across 10 NVIDIA Graphics Processing Units (GPUs) in batch and server scenarios along with a reproducible, open-source benchmark that enables community submissions to expand this dataset. Leveraging this dataset, we conduct a system-level study of LLM inference across heterogeneous GPU architectures and show that GPU selection is crucial for energy efficiency outcomes and that optimal hardware choices vary significantly across models and deployment scenarios, demonstrating the critical importance of hardware-aware deployment in heterogeneous LLM systems. Guided by our data and insights, we show that practitioners can reduce energy consumption by up to 70% in server scenarios with negligible impact on user experience, and by up to 20% in batch scenarios.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 44 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

25 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving5 Aug

Other references · 24

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  4. 2019MLPerf Inference Benchmarkcited by 195
  5. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 168
  6. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  7. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  8. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  9. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 78
  10. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 64
  11. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 54
  12. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 32
12 more
  1. 2025The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimizationcited by 31
  2. 2025Holistically Evaluating the Environmental Impact of Creating Language Modelscited by 26
  3. 2024The Price of Prompting: Profiling Energy Use in Large Language Models Inferencecited by 22
  4. 2025TokenPowerBench: Benchmarking the Power Consumption of LLM Inferencecited by 21
  5. 2024LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferencescited by 19
  6. 2024LLMCBench: Benchmarking Large Language Model Compression for Efficient Deploymentcited by 16
  7. 2023Part-time Power Measurements: nvidia-smi's Lack of Attentioncited by 15
  8. 2024Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Trainingcited by 15
  9. 2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 15
  10. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 14
  11. 2025A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiencycited by 12
  12. 2025Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMscited by 1
  2. 2026TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Servingcited by 0
  3. 2026Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scalingcited by 0
  4. 2026Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inferencecited by 0
  5. 2026The Right Information Extraction Pipeline Depends on the Document: Accuracy-Energy Trade-offs for Small, Local Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization
  2. 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
  3. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  4. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems
  5. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference
  6. 2024GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions
  7. 2026PALS: Power-Aware LLM Serving for Mixture-of-Experts Models
  8. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs