# The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization

arXiv: https://arxiv.org/abs/2505.06371 · HTML: https://mlsanity.com/paper/2505.06371/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 77 references that are in the map.
- First posted on arXiv 9 May 2025.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)

## Built on by (most cited first, 30 of 33)

- [Measuring the environmental impact of delivering AI at Google Scale](https://mlsanity.com/paper/2508.15734/) (2025)
- [Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms](https://mlsanity.com/paper/2508.18298/) (2025)
- [Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling](https://mlsanity.com/paper/2509.20241/) (2025)
- [Where Do the Joules Go? Diagnosing Inference Energy Consumption](https://mlsanity.com/paper/2601.22076/) (2026)
- [Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures](https://mlsanity.com/paper/2604.09048/) (2026)
- [Bench360: Benchmarking Local LLM Inference from 360 Degrees](https://mlsanity.com/paper/2511.16682/) (2025)
- [Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training](https://mlsanity.com/paper/2601.17654/) (2026)
- [ELANA: A Simple Energy and Latency Analyzer for LLMs](https://mlsanity.com/paper/2512.09946/) (2025)
- [GELATO: Generative Entropy- and Lyapunov-based Adaptive Token Offloading for Device-Edge Speculative LLM Inference](https://mlsanity.com/paper/2605.10124/) (2026)
- [Energy per Successful Goal: Goal-Level Energy Accounting for Agentic AI Systems](https://mlsanity.com/paper/2605.22883/) (2026)
- [Comparing energy consumption and accuracy in text classification inference](https://mlsanity.com/paper/2508.14170/) (2025)
- [From Independent to Correlated Diffusion: Generalized Generative Modeling with Probabilistic Computers](https://mlsanity.com/paper/2603.27996/) (2026)
- [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](https://mlsanity.com/paper/2604.16682/) (2026)
- [BIRDS: Characterizing and Understanding Biodiversity Impact of Large Language Model Serving](https://mlsanity.com/paper/2605.27480/) (2026)
- [WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs](https://mlsanity.com/paper/2607.02391/) (2026)
- [Request-Level Energy Attribution for Batched LLM Serving](https://mlsanity.com/paper/2608.00026/) (2026)
- [The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations](https://mlsanity.com/paper/2507.13302/) (2025)
- [Magneton: Optimizing Energy Efficiency of ML Systems via Differential Energy Debugging](https://mlsanity.com/paper/2512.08365/) (2025)
- [Balancing Sustainability And Performance: The Role Of Small-Scale LLMs In Agentic Artificial Intelligence Systems](https://mlsanity.com/paper/2601.19311/) (2026)
- [Transparent Screening for LLM Inference and Training Impacts](https://mlsanity.com/paper/2604.19757/) (2026)
- [OpenG2G: A Simulation Platform for AI Datacenter-Grid Runtime Coordination](https://mlsanity.com/paper/2605.05519/) (2026)
- [The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inference](https://mlsanity.com/paper/2606.21869/) (2026)
- [A Picture is Worth a Thousand Tokens: How Vision Language Models Cut AI Energy Costs While Improving Accuracy](https://mlsanity.com/paper/2608.07427/) (2026)
- [InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centers](https://mlsanity.com/paper/2608.12915/) (2026)
- [TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Serving](https://mlsanity.com/paper/2608.18149/) (2026)
- [Vision-centric generative AI models: A software-hardware perspective](https://mlsanity.com/paper/2608.27199/) (2026)
- [Characterizing Job Power Elasticity for Power-Flexible AI Training](https://mlsanity.com/paper/2609.11542/) (2026)
- [Where Should Agents Live? Energy-Memory Characterization of Agentic AI for the Edge-Cloud Continuum](https://mlsanity.com/paper/2609.18283/) (2026)
- [Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving](https://mlsanity.com/paper/2609.23085/) (2026)
- [Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Models](https://mlsanity.com/paper/2609.33965/) (2026)

## Abstract (arXiv)

As the adoption of Generative AI in real-world services grow explosively, energy has emerged as a critical bottleneck resource. However, energy remains a metric that is often overlooked, under-explored, or poorly understood in the context of building ML systems. We present the ML.ENERGY Benchmark, a benchmark suite and tool for measuring inference energy consumption under realistic service environments, and the corresponding ML.ENERGY Leaderboard, which have served as a valuable resource for those hoping to understand and optimize the energy consumption of their generative AI services. In this paper, we explain four key design principles for benchmarking ML energy we have acquired over time, and then describe how they are implemented in the ML.ENERGY Benchmark. We then highlight results from the early 2025 iteration of the benchmark, including energy measurements of 40 widely used model architectures across 6 different tasks, case studies of how ML design choices impact energy consumption, and how automated optimization recommendations can lead to significant (sometimes more than 40%) energy savings without changing what is being computed by the model. The ML.ENERGY Benchmark is open-source and can be easily extended to various customized models and application scenarios.