# Energy Considerations of Large Language Model Inference and Efficiency Optimizations

arXiv: https://arxiv.org/abs/2504.17674 · HTML: https://mlsanity.com/paper/2504.17674/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 56 references that are in the map.
- First posted on arXiv 24 Apr 2025.

## Builds on (most cited first, 27 of 27)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink](https://mlsanity.com/paper/2204.05149/) (2022)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding](https://mlsanity.com/paper/2408.11049/) (2024)
- [BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems](https://mlsanity.com/paper/2401.17644/) (2024)
- [Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference](https://mlsanity.com/paper/2403.20306/) (2024)
- [Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems](https://mlsanity.com/paper/2407.04014/) (2024)
- [Holistically Evaluating the Environmental Impact of Creating Language Models](https://mlsanity.com/paper/2503.05804/) (2025)
- [Energy and Carbon Considerations of Fine-Tuning BERT](https://mlsanity.com/paper/2311.10267/) (2023)
- [Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit View](https://mlsanity.com/paper/2502.11256/) (2025)
- [GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions](https://mlsanity.com/paper/2412.20322/) (2024)

## Built on by (most cited first, 30 of 55)

- [A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services](https://mlsanity.com/paper/2509.18101/) (2025)
- [The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute](https://mlsanity.com/paper/2505.14733/) (2025)
- [Intelligence per Watt: Measuring Intelligence Efficiency of Local AI](https://mlsanity.com/paper/2511.07885/) (2025)
- [Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives](https://mlsanity.com/paper/2505.21627/) (2025)
- [OckBench: Measuring the Efficiency of LLM Reasoning](https://mlsanity.com/paper/2511.05722/) (2025)
- [Does quantization affect models' performance on long-context tasks?](https://mlsanity.com/paper/2505.20276/) (2025)
- [Efficient Reasoning Through Suppression of Self-Affirmation Reflections in Large Reasoning Models](https://mlsanity.com/paper/2506.12353/) (2025)
- [Layer-wise dynamic rank for compressing large language models](https://mlsanity.com/paper/2509.25622/) (2025)
- [SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inference](https://mlsanity.com/paper/2602.05695/) (2026)
- [Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures](https://mlsanity.com/paper/2604.09048/) (2026)
- [ECVL-ROUTER: Scenario-Aware Routing for Vision-Language Models](https://mlsanity.com/paper/2510.27256/) (2025)
- [Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use](https://mlsanity.com/paper/2601.22362/) (2026)
- [This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMs](https://mlsanity.com/paper/2603.15699/) (2026)
- [From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs](https://mlsanity.com/paper/2607.26571/) (2026)
- [GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Model](https://mlsanity.com/paper/2508.16700/) (2025)
- [VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing](https://mlsanity.com/paper/2509.04827/) (2025)
- [Towards Green AI: Decoding the Energy of LLM Inference in Software Development](https://mlsanity.com/paper/2602.05712/) (2026)
- [Resource Consumption Threats in Large Language Models](https://mlsanity.com/paper/2603.16068/) (2026)
- [Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows](https://mlsanity.com/paper/2604.09611/) (2026)
- [Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms](https://mlsanity.com/paper/2608.28044/) (2026)
- [Dissecting Transformers: A CLEAR Perspective towards Green AI](https://mlsanity.com/paper/2510.02810/) (2025)
- [Performance Trade-offs of Optimizing Small Language Models for E-Commerce](https://mlsanity.com/paper/2510.21970/) (2025)
- [Big AI is accelerating the metacrisis: What can we do?](https://mlsanity.com/paper/2512.24863/) (2025)
- [Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP System](https://mlsanity.com/paper/2601.05267/) (2025)
- [Small Talk, Big Impact: The Energy Cost of Thanking AI](https://mlsanity.com/paper/2601.22357/) (2026)
- [iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems](https://mlsanity.com/paper/2602.06064/) (2026)
- [Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameters](https://mlsanity.com/paper/2602.17697/) (2026)
- [Generative Modeling in Protein Design: Neural Representations, Conditional Generation, and Evaluation Standards](https://mlsanity.com/paper/2603.26378/) (2026)
- [SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework](https://mlsanity.com/paper/2607.00274/) (2026)
- [Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations](https://mlsanity.com/paper/2607.09172/) (2026)

## Abstract (arXiv)

As large language models (LLMs) scale in size and adoption, their computational and environmental costs continue to rise. Prior benchmarking efforts have primarily focused on latency reduction in idealized settings, often overlooking the diverse real-world inference workloads that shape energy use. In this work, we systematically analyze the energy implications of common inference efficiency optimizations across diverse Natural Language Processing (NLP) and generative Artificial Intelligence (AI) workloads, including conversational AI and code generation. We introduce a modeling approach that approximates real-world LLM workflows through a binning strategy for input-output token distributions and batch size variations. Our empirical analysis spans software frameworks, decoding strategies, GPU architectures, online and offline serving settings, and model parallelism configurations. We show that the effectiveness of inference optimizations is highly sensitive to workload geometry, software stack, and hardware accelerators, demonstrating that naive energy estimates based on FLOPs or theoretical GPU utilization significantly underestimate real-world energy consumption. Our findings reveal that the proper application of relevant inference efficiency optimizations can reduce total energy use by up to 73% from unoptimized baselines. These insights provide a foundation for sustainable LLM deployment and inform energy-efficient design strategies for future AI infrastructure.