mlsanityData to 14 Oct 2026

The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization

arXiv 2505.06371 ↗First posted 9 May 2025cs.LG cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

As the adoption of Generative AI in real-world services grow explosively, energy has emerged as a critical bottleneck resource. However, energy remains a metric that is often overlooked, under-explored, or poorly understood in the context of building ML systems. We present the ML.ENERGY Benchmark, a benchmark suite and tool for measuring inference energy consumption under realistic service environments, and the corresponding ML.ENERGY Leaderboard, which have served as a valuable resource for those hoping to understand and optimize the energy consumption of their generative AI services. In this paper, we explain four key design principles for benchmarking ML energy we have acquired over time, and then describe how they are implemented in the ML.ENERGY Benchmark. We then highlight results from the early 2025 iteration of the benchmark, including energy measurements of 40 widely used model architectures across 6 different tasks, case studies of how ML design choices impact energy consumption, and how automated optimization recommendations can lead to significant (sometimes more than 40%) energy savings without changing what is being computed by the model. The ML.ENERGY Benchmark is open-source and can be easily extended to various customized models and application scenarios.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 45 of its 77 references that are in the map. First posted on arXiv 9 May 2025.

Builds on

45 of 77 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
33 more
  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2024Mixtral of Expertscited by 1,823
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  11. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  12. 2023ModelScope Text-to-Video Technical Reportcited by 658
  13. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  14. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  15. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  16. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  17. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  18. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  19. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  20. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  21. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  22. 2025Phi-4-reasoning Technical Reportcited by 151
  23. 2024CodeGemma: Open Code Models Based on Gemmacited by 150
  24. 2023Dual Aggregation Transformer for Image Super-Resolutioncited by 124
  25. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  26. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  27. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 79
  28. 2024Progressive Knowledge Distillation Of Stable Diffusion XL Using Layer Level Losscited by 52
  29. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45
  30. 2023Reducing Energy Bloat in Large Model Trainingcited by 22
  31. 2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 16
  32. 2020Verified Instruction-Level Energy Consumption Measurement for NVIDIA GPUscited by 10
  33. 2024Toward Cross-Layer Energy Optimizations in AI Systemscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 25.

33 papers.

Sort:

By area:

  1. 2025Measuring the environmental impact of delivering AI at Google Scalecited by 32
  2. 2025Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platformscited by 18
  3. 2025Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scalingcited by 13
  4. 2026Where Do the Joules Go? Diagnosing Inference Energy Consumptioncited by 10
  5. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architecturescited by 5
  6. 2025Bench360: Benchmarking Local LLM Inference from 360 Degreescited by 4
  7. 2026Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Trainingcited by 3
  8. 2025ELANA: A Simple Energy and Latency Analyzer for LLMscited by 2
  9. 2026GELATO: Generative Entropy- and Lyapunov-based Adaptive Token Offloading for Device-Edge Speculative LLM Inferencecited by 2
  10. 2026Energy per Successful Goal: Goal-Level Energy Accounting for Agentic AI Systemscited by 2
  11. 2025Comparing energy consumption and accuracy in text classification inferencecited by 1
  12. 2026From Independent to Correlated Diffusion: Generalized Generative Modeling with Probabilistic Computerscited by 1
21 more
  1. 2026KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Servingcited by 1
  2. 2026BIRDS: Characterizing and Understanding Biodiversity Impact of Large Language Model Servingcited by 1
  3. 2026WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMscited by 1
  4. 2026Request-Level Energy Attribution for Batched LLM Servingcited by 1
  5. 2025The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluationscited by 0
  6. 2025Magneton: Optimizing Energy Efficiency of ML Systems via Differential Energy Debuggingcited by 0
  7. 2026Balancing Sustainability And Performance: The Role Of Small-Scale LLMs In Agentic Artificial Intelligence Systemscited by 0
  8. 2026Transparent Screening for LLM Inference and Training Impactscited by 0
  9. 2026OpenG2G: A Simulation Platform for AI Datacenter-Grid Runtime Coordinationcited by 0
  10. 2026The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inferencecited by 0
  11. 2026A Picture is Worth a Thousand Tokens: How Vision Language Models Cut AI Energy Costs While Improving Accuracycited by 0
  12. 2026InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centerscited by 0
  13. 2026TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Servingcited by 0
  14. 2026Vision-centric generative AI models: A software-hardware perspectivecited by 0
  15. 2026Characterizing Job Power Elasticity for Power-Flexible AI Trainingcited by 0
  16. 2026Where Should Agents Live? Energy-Memory Characterization of Agentic AI for the Edge-Cloud Continuumcited by 0
  17. 2026Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Servingcited by 0
  18. 2026Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Modelscited by 0
  19. 2026AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMscited by 0
  20. 2026Beyond Energy: When Sustainability Dimensions Reshape LLM Serving Decisionscited by 0
  21. 2026When Is Smaller Greener? Right-Sizing Language Models for Agentic AIcited by 0