mlsanityData to 14 Oct 2026

Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling

arXiv 2509.20241 ↗First posted 24 Sep 2025cs.LG cs.DCcited by 13

Not on the map. Why?

Abstract (arXiv)

As AI inference scales to billions of queries, estimates of per-query energy use are increasingly important for capacity planning, efficiency interventions, and policy. Yet many public estimates assume non-production settings, leading to systematic overestimation. We introduce a bottom-up framework estimating inference energy from token throughput, node power, and overhead under large-scale deployment assumptions. For frontier-scale models (>200B parameters) on H100 nodes, we estimate a median energy of 0.31 Wh/query (IQR 0.16-0.60), indicating widely cited estimates are overstated by 4-20x. In test-time scaling scenarios 15x longer than typical queries, the median energy rises 13x to 3.91 Wh (IQR 2.15-7.05). Across models, serving systems, and hardware, we estimate 8-20x line-of-sight energy reductions. At datacenter scale, serving 1 billion queries/day requires 0.7 GWh; if 10% are long queries, demand rises to 1.7 GWh/day. With efficiency interventions, it falls to 0.8 GWh/day, mitigating the energy impact of test-time scaling.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 36 of its 64 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

36 of 64 references.

References · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025Gemma 3 Technical Reportcited by 1,824
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  5. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  10. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  11. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  12. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
24 more
  1. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  2. 2025Deep Think with Confidencecited by 205
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  4. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
  5. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
  6. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 136
  7. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  8. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  9. 2025How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inferencecited by 76
  10. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  11. 2020Chasing Carbon: The Elusive Environmental Footprint of Computingcited by 72
  12. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  13. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
  14. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  15. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 52
  16. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
  17. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  18. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  19. 2025The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimizationcited by 33
  20. 2025Measuring the environmental impact of delivering AI at Google Scalecited by 32
  21. 2025From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debatecited by 23
  22. 2025EcoServe: Designing Carbon-Aware AI Inference Systemscited by 19
  23. 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Computecited by 15
  24. 2025Comparing energy consumption and accuracy in text classification inferencecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 14
  2. 2026Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observabilitycited by 4
  3. 2026Energy per Successful Goal: Goal-Level Energy Accounting for Agentic AI Systemscited by 2
  4. 2026PANDO: Efficient Multimodal AI Agents via Online Skill Distillationcited by 2
  5. 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platformscited by 2
  6. 2026Evaluation of ML Resource Utilization Requires Model Life Cycle Assessmentcited by 0
  7. 2026Quantifying the Relationship Between Clinical Safety and Environmental Impact in Therapeutic LLMscited by 0
  8. 2026From Caveman to Expert Analyst: Energy Consumption of Variable LLM Taskscited by 0
  9. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inferencecited by 0
  10. 2026Where Should Agents Live? Energy-Memory Characterization of Agentic AI for the Edge-Cloud Continuumcited by 0
  11. 2026Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Modelscited by 0
  12. 2026Enterprise Representation Simplification (ERS): Reducing Representational Complexity for Enterprise AIcited by 0
1 more
  1. 2026When Is Smaller Greener? Right-Sizing Language Models for Agentic AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Position: LLM Inference Should Be Evaluated as Energy-to-Token Production
  2. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
  3. 2026Coordinated Cooling and Compute Management for AI Datacenters ↗
  4. 2026Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware
  5. 2026Where Does the Energy Go? Profiling LLM Agent Inference on Blackwell GPUs ↗
  6. 2026Characterizing Job Power Elasticity for Power-Flexible AI Training
  7. 2026A Picture is Worth a Thousand Tokens: How Vision Language Models Cut AI Energy Costs While Improving Accuracy
  8. 2026XWind: A Cross-site Router for Large Language Model Inference Serving at Renewable Energy Farms