# Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference

arXiv: https://arxiv.org/abs/2403.20306 · HTML: https://mlsanity.com/paper/2403.20306/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills](https://mlsanity.com/paper/2308.16369/) (2023)
- [LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://mlsanity.com/paper/2312.11514/) (2023)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [Chasing Carbon: The Elusive Environmental Footprint of Computing](https://mlsanity.com/paper/2011.02839/) (2020)
- [LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models](https://mlsanity.com/paper/2309.14393/) (2023)
- [Characterization of Large Language Model Development in the Datacenter](https://mlsanity.com/paper/2403.07648/) (2024)
- [Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey](https://mlsanity.com/paper/2203.08737/) (2022)
- [NPE: An FPGA-based Overlay Processor for Natural Language Processing](https://mlsanity.com/paper/2104.06535/) (2021)
- [A Hardware Evaluation Framework for Large Language Model Inference](https://mlsanity.com/paper/2312.03134/) (2023)
- [POLCA: Power Oversubscription in LLM Cloud Providers](https://mlsanity.com/paper/2308.12908/) (2023)

## Built on by (most cited first, 30 of 61)

- [DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency](https://mlsanity.com/paper/2408.00741/) (2024)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [Energy Considerations of Large Language Model Inference and Efficiency Optimizations](https://mlsanity.com/paper/2504.17674/) (2025)
- [A Survey of Small Language Models](https://mlsanity.com/paper/2410.20011/) (2024)
- [TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms](https://mlsanity.com/paper/2501.02600/) (2025)
- [Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems](https://mlsanity.com/paper/2407.04014/) (2024)
- [InfiniPot: Infinite Context Processing on Memory-Constrained LLMs](https://mlsanity.com/paper/2410.01518/) (2024)
- [The Price of Prompting: Profiling Energy Use in Large Language Models Inference](https://mlsanity.com/paper/2407.16893/) (2024)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time](https://mlsanity.com/paper/2408.13233/) (2024)
- [SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving](https://mlsanity.com/paper/2408.05235/) (2024)
- [An overview of domain-specific foundation model: key technologies, applications and challenges](https://mlsanity.com/paper/2409.04267/) (2024)
- [MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI](https://mlsanity.com/paper/2410.12032/) (2024)
- [The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute](https://mlsanity.com/paper/2505.14733/) (2025)
- [Accelerating Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.15246/) (2024)
- [From Prompts to Power: Measuring the Energy Footprint of LLM Inference](https://mlsanity.com/paper/2511.05597/) (2025)
- [KV Prediction for Improved Time to First Token](https://mlsanity.com/paper/2410.08391/) (2024)
- [Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments](https://mlsanity.com/paper/2411.17741/) (2024)
- [Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures](https://mlsanity.com/paper/2504.11750/) (2025)
- [Successor-Generator Planning with LLM-generated Heuristics](https://mlsanity.com/paper/2501.18784/) (2025)
- [SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inference](https://mlsanity.com/paper/2602.05695/) (2026)
- [Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques](https://mlsanity.com/paper/2506.06579/) (2025)
- [DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS](https://mlsanity.com/paper/2602.18755/) (2026)
- [All-in-One Tuning and Structural Pruning for Domain-Specific LLMs](https://mlsanity.com/paper/2412.14426/) (2024)
- [Improving training time and GPU utilization in geo-distributed language model training](https://mlsanity.com/paper/2411.14458/) (2024)
- [CodEv: An Automated Grading Framework Leveraging Large Language Models for Consistent and Constructive Feedback](https://mlsanity.com/paper/2501.10421/) (2025)
- [Green Prompting: Characterizing Prompt-driven Energy Costs of LLM Inference](https://mlsanity.com/paper/2503.10666/) (2025)
- [Matmul or No Matmul in the Era of 1-bit LLMs](https://mlsanity.com/paper/2408.11939/) (2024)
- [Continuous Approximations for Improving Quantization Aware Training of LLMs](https://mlsanity.com/paper/2410.10849/) (2024)
- [MedSlice: Fine-Tuned Large Language Models for Secure Clinical Note Sectioning](https://mlsanity.com/paper/2501.14105/) (2025)

## Abstract (arXiv)

With the ubiquitous use of modern large language models (LLMs) across industries, the inference serving for these models is ever expanding. Given the high compute and memory requirements of modern LLMs, more and more top-of-the-line GPUs are being deployed to serve these models. Energy availability has come to the forefront as the biggest challenge for data center expansion to serve these models. In this paper, we present the trade-offs brought up by making energy efficiency the primary goal of LLM serving under performance SLOs. We show that depending on the inputs, the model, and the service-level agreements, there are several knobs available to the LLM inference provider to use for being energy efficient. We characterize the impact of these knobs on the latency, throughput, as well as the energy. By exploring these trade-offs, we offer valuable insights into optimizing energy usage without compromising on performance, thereby paving the way for sustainable and cost-effective LLM deployment in data center environments.