mlsanityData to 7 Oct 2026

Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference

arXiv 2403.20306 ↗First posted 29 Mar 2024cs.AI cs.AR cs.DCcited by 59

Not on the map. Why?

Abstract (arXiv)

With the ubiquitous use of modern large language models (LLMs) across industries, the inference serving for these models is ever expanding. Given the high compute and memory requirements of modern LLMs, more and more top-of-the-line GPUs are being deployed to serve these models. Energy availability has come to the forefront as the biggest challenge for data center expansion to serve these models. In this paper, we present the trade-offs brought up by making energy efficiency the primary goal of LLM serving under performance SLOs. We show that depending on the inputs, the model, and the service-level agreements, there are several knobs available to the LLM inference provider to use for being energy efficient. We characterize the impact of these knobs on the latency, throughput, as well as the energy. By exploring these trade-offs, we offer valuable insights into optimizing energy usage without compromising on performance, thereby paving the way for sustainable and cost-effective LLM deployment in data center environments.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 20 of its 40 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

20 of 40 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2023Textbooks Are All You Needcited by 561
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  9. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  10. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  11. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  12. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
8 more
  1. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  2. 2020Chasing Carbon: The Elusive Environmental Footprint of Computingcited by 71
  3. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Modelscited by 71
  4. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  5. 2022Hardware Approximate Techniques for Deep Neural Network Accelerators: A Surveycited by 39
  6. 2021NPE: An FPGA-based Overlay Processor for Natural Language Processingcited by 15
  7. 2023A Hardware Evaluation Framework for Large Language Model Inferencecited by 14
  8. 2023POLCA: Power Oversubscription in LLM Cloud Providerscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 25; 2026: 16.

59 papers.

Sort:

By area:

  1. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  2. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  3. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 54
  4. 2024A Survey of Small Language Modelscited by 31
  5. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  6. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systemscited by 27
  7. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  8. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  9. 2024The Price of Prompting: Profiling Energy Use in Large Language Models Inferencecited by 22
  10. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  11. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
  12. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
38 more
  1. 2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 15
  2. 2024Accelerating Retrieval-Augmented Generationcited by 14
  3. 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Computecited by 14
  4. 2025From Prompts to Power: Measuring the Energy Footprint of LLM Inferencecited by 12
  5. 2024KV Prediction for Improved Time to First Tokencited by 9
  6. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
  7. 2025Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architecturescited by 8
  8. 2025Successor-Generator Planning with LLM-generated Heuristicscited by 7
  9. 2026SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inferencecited by 6
  10. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  11. 2024All-in-One Tuning and Structural Pruning for Domain-Specific LLMscited by 4
  12. 2026DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFScited by 4
  13. 2024Improving training time and GPU utilization in geo-distributed language model trainingcited by 3
  14. 2025CodEv: An Automated Grading Framework Leveraging Large Language Models for Consistent and Constructive Feedbackcited by 3
  15. 2025Green Prompting: Characterizing Prompt-driven Energy Costs of LLM Inferencecited by 3
  16. 2024Matmul or No Matmul in the Era of 1-bit LLMscited by 2
  17. 2024Continuous Approximations for Improving Quantization Aware Training of LLMscited by 2
  18. 2025MedSlice: Fine-Tuned Large Language Models for Secure Clinical Note Sectioningcited by 2
  19. 2025VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routingcited by 2
  20. 2025Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Frameworkcited by 2
  21. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  22. 2026Towards Green AI: Decoding the Energy of LLM Inference in Software Developmentcited by 2
  23. 2026Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflowscited by 2
  24. 2025Good things come in small packages: Should we build AI clusters with Lite-GPUs?cited by 1
  25. 2025Developing Conversational Speech Systems for Robots to Detect Speech Biomarkers of Cognition in People Living with Dementiacited by 1
  26. 2025PIM-LLM: A High-Throughput Hybrid PIM Architecture for 1-bit LLMscited by 1
  27. 2025Prompting Large Language Models for Training-Free Non-Intrusive Load Monitoringcited by 1
  28. 2025FrugalPrompt: Reducing Contextual Overhead in Large Language Models via Token Attributioncited by 1
  29. 2025Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP Systemcited by 1
  30. 2026Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameterscited by 1
  31. 2026KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Servingcited by 1
  32. 2026EnerInfer: Energy-Aware On-Device LLM Inferencecited by 1
  33. 2026Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurationscited by 1
  34. 2024Software Performance Engineering for Foundation Model-Powered Softwarecited by 0
  35. 2025Toward Neurosymbolic Program Comprehensioncited by 0
  36. 2025MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systemscited by 0
  37. 2025Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptationcited by 0
  38. 2025Is There a Case for Conversation Optimized Tokenizers in Large Language Models?cited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
  2. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  3. 2026PALS: Power-Aware LLM Serving for Mixture-of-Experts Models
  4. 2026EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization
  5. 2026Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
  6. 2026WANSpec: Leveraging Global Compute Capacity for LLM Inference
  7. 2024Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching
  8. 2026Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference