mlsanityData to 14 Oct 2026

Energy Considerations of Large Language Model Inference and Efficiency Optimizations

arXiv 2504.17674 ↗First posted 24 Apr 2025cs.CL cs.LGcited by 55

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) scale in size and adoption, their computational and environmental costs continue to rise. Prior benchmarking efforts have primarily focused on latency reduction in idealized settings, often overlooking the diverse real-world inference workloads that shape energy use. In this work, we systematically analyze the energy implications of common inference efficiency optimizations across diverse Natural Language Processing (NLP) and generative Artificial Intelligence (AI) workloads, including conversational AI and code generation. We introduce a modeling approach that approximates real-world LLM workflows through a binning strategy for input-output token distributions and batch size variations. Our empirical analysis spans software frameworks, decoding strategies, GPU architectures, online and offline serving settings, and model parallelism configurations. We show that the effectiveness of inference optimizations is highly sensitive to workload geometry, software stack, and hardware accelerators, demonstrating that naive energy estimates based on FLOPs or theoretical GPU utilization significantly underestimate real-world energy consumption. Our findings reveal that the proper application of relevant inference efficiency optimizations can reduce total energy use by up to 73% from unoptimized baselines. These insights provide a foundation for sustainable LLM deployment and inform energy-efficient design strategies for future AI infrastructure.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 27 of its 56 references that are in the map. First posted on arXiv 24 Apr 2025.

Builds on

27 of 56 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  12. 2019Green AIcited by 627
15 more
  1. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  4. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  5. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 191
  6. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
  7. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
  8. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 91
  9. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 87
  10. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
  11. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systemscited by 28
  12. 2025Holistically Evaluating the Environmental Impact of Creating Language Modelscited by 26
  13. 2023Energy and Carbon Considerations of Fine-Tuning BERTcited by 11
  14. 2025Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit Viewcited by 11
  15. 2024GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissionscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 38.

55 papers.

Sort:

By area:

  1. 2025A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Servicescited by 17
  2. 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Computecited by 15
  3. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 14
  4. 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
  5. 2025OckBench: Measuring the Efficiency of LLM Reasoningcited by 11
  6. 2025Does quantization affect models' performance on long-context tasks?cited by 10
  7. 2025Efficient Reasoning Through Suppression of Self-Affirmation Reflections in Large Reasoning Modelscited by 10
  8. 2025Layer-wise dynamic rank for compressing large language modelscited by 8
  9. 2026SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inferencecited by 6
  10. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architecturescited by 5
  11. 2025ECVL-ROUTER: Scenario-Aware Routing for Vision-Language Modelscited by 4
  12. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Usecited by 4
38 more
  1. 2026This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMscited by 3
  2. 2026From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUscited by 3
  3. 2025GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Modelcited by 2
  4. 2025VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routingcited by 2
  5. 2026Towards Green AI: Decoding the Energy of LLM Inference in Software Developmentcited by 2
  6. 2026Resource Consumption Threats in Large Language Modelscited by 2
  7. 2026Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflowscited by 2
  8. 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platformscited by 2
  9. 2025Dissecting Transformers: A CLEAR Perspective towards Green AIcited by 1
  10. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  11. 2025Big AI is accelerating the metacrisis: What can we do?cited by 1
  12. 2025Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP Systemcited by 1
  13. 2026Small Talk, Big Impact: The Energy Cost of Thanking AIcited by 1
  14. 2026iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problemscited by 1
  15. 2026Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameterscited by 1
  16. 2026Generative Modeling in Protein Design: Neural Representations, Conditional Generation, and Evaluation Standardscited by 1
  17. 2026SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Frameworkcited by 1
  18. 2026Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurationscited by 1
  19. 2025Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-servicecited by 0
  20. 2025Enabling Physical AI at the Edge: Hardware-Accelerated Recovery of System Dynamicscited by 0
  21. 2026Is Knowledge Distillation Actually Greener? A Case Study in Machine Translationcited by 0
  22. 2026Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guaranteescited by 0
  23. 2026SEALing the Gap: A Reference Framework for LLM Inference Carbon Estimation via Multi-Benchmark Driven Embodimentcited by 0
  24. 2026Energy Efficient Software Hardware CoDesign for Machine Learning: From TinyML to Large Language Modelscited by 0
  25. 2026Cross-Family Speculative Decoding for Polish Language Models on Apple~Silicon: An Empirical Evaluation of Bielik~11B with UAG-Extended MLX-LMcited by 0
  26. 2026Long Live the Librarian! A Persistent Search Sub-Agent for Energy-Efficient Multi-Agent Software Engineering Systemscited by 0
  27. 2026Evaluation of ML Resource Utilization Requires Model Life Cycle Assessmentcited by 0
  28. 2026From Perception to Action: Can UI Interventions Foster Sustainable LLM Chatbotcited by 0
  29. 2026Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extractioncited by 0
  30. 2026Lights, Camera, Carbon: Architectural Scaling Laws for Video Generation Energy Consumptioncited by 0
  31. 2026Human Grounded Evaluation of Large Language Models for Optical Network Automationcited by 0
  32. 2026Unified Static-Dynamic Pruning for Efficient LLM Inferencecited by 0
  33. 2026ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentationcited by 0
  34. 2026From Caveman to Expert Analyst: Energy Consumption of Variable LLM Taskscited by 0
  35. 2026TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Servingcited by 0
  36. 2026Artificial Intelligence for Energy Optimization in Data Centerscited by 0
  37. 2026Toward Sustainable Distributed LLM Inference: A Systems Synthesis and Research Agenda for an Energy-, Carbon-, and Cache-Aware llm-d Control Planecited by 0
  38. 2026Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scalingcited by 0

50 most cited of 55 All 55 (JSON)