# Where Do the Joules Go? Diagnosing Inference Energy Consumption

arXiv: https://arxiv.org/abs/2601.22076 · HTML: https://mlsanity.com/paper/2601.22076/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers](https://mlsanity.com/paper/2410.10629/) (2024)
- [PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation](https://mlsanity.com/paper/2403.04692/) (2024)
- [DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency](https://mlsanity.com/paper/2408.00741/) (2024)
- [NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model](https://mlsanity.com/paper/2508.14444/) (2025)
- [Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training](https://mlsanity.com/paper/2208.06102/) (2022)
- [xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism](https://mlsanity.com/paper/2411.01738/) (2024)
- [The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization](https://mlsanity.com/paper/2505.06371/) (2025)
- [Measuring the environmental impact of delivering AI at Google Scale](https://mlsanity.com/paper/2508.15734/) (2025)
- [Reducing Energy Bloat in Large Model Training](https://mlsanity.com/paper/2312.06902/) (2023)
- [MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI](https://mlsanity.com/paper/2410.12032/) (2024)
- [Electricity Demand and Grid Impacts of AI Data Centers: Challenges and Prospects](https://mlsanity.com/paper/2509.07218/) (2025)
- [Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training](https://mlsanity.com/paper/2601.17654/) (2026)

## Built on by (most cited first, 10 of 10)

- [Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference](https://mlsanity.com/paper/2607.09520/) (2026)
- [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](https://mlsanity.com/paper/2604.16682/) (2026)
- [OpenG2G: A Simulation Platform for AI Datacenter-Grid Runtime Coordination](https://mlsanity.com/paper/2605.05519/) (2026)
- [The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inference](https://mlsanity.com/paper/2606.21869/) (2026)
- [The Many-Body Problem of the Data Centre](https://mlsanity.com/paper/2606.30206/) (2026)
- [Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence](https://mlsanity.com/paper/2607.20981/) (2026)
- [Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures](https://mlsanity.com/paper/2608.25096/) (2026)
- [Where Should Agents Live? Energy-Memory Characterization of Agentic AI for the Edge-Cloud Continuum](https://mlsanity.com/paper/2609.18283/) (2026)
- [Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving](https://mlsanity.com/paper/2609.23085/) (2026)
- [Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Models](https://mlsanity.com/paper/2609.33965/) (2026)

## Abstract (arXiv)

Energy is now a critical ML computing resource. While measuring energy consumption and observing trends is a valuable first step, accurately understanding and diagnosing why those differences occur is crucial for optimization. To that end, we begin by presenting a large-scale measurement study of inference time and energy across the generative AI landscape with 46 models, 7 tasks, and 1,858 different configurations on NVIDIA H100 and B200 GPUs. Our empirical findings span order-of-magnitude variations: LLM task type can lead to 25× energy differences, video generation sometimes consumes more than 100× the energy of images, and GPU utilization differences can result in 3--5× energy differences. Based on our observations, we present a framework for reasoning about the underlying mechanisms that govern time and energy consumption. The essence is that time and energy are determined by latent metrics like memory and utilization, which are in turn affected by various factors across the algorithm, software, and hardware layers. Our framework also extends directly to throughput per watt, a critical metric for power-constrained datacenters.