Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are increasingly recognized for their exceptional generative capabilities and versatility across various tasks. However, the high inference costs associated with these models have not received adequate attention, particularly when compared to the focus on training costs in existing research. In response to this gap, our study conducts a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks, where we analyze the impact of different models, tasks, prompts, and system-related factors on inference energy. Specifically, our experiments reveal several interesting insights, including strong correlation of inference energy with output token length and response time. Also, we find that quantization and optimal batch sizes, along with targeted prompt phrases, can significantly reduce energy usage. This study is the first to thoroughly benchmark LLM inference across such a diverse range of aspects, providing insights and offering several recommendations for improving energy efficiency in model deployment.
Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 8 Feb 2025.
Builds on
33 of 51 references.
References · 33
- 2017Attention Is All You Needcited by 60,361
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
- 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
- 2019Green AIcited by 627
21 more
- 2021Carbon Emissions and Large Neural Network Trainingcited by 567
- 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
- 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
- 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 249
- 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
- 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 191
- 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
- 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
- 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 169
- 2022Measuring the Carbon Intensity of AI in Cloud Instancescited by 111
- 2021Compute and Energy Consumption Trends in Deep Learning Inferencecited by 61
- 2022Eco2AI: carbon emissions tracking of machine learning models as the first step towards sustainable AIcited by 58
- 2023Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learningcited by 42
- 2023How to estimate carbon footprint when training deep learning models? A guide and reviewcited by 34
- 2022Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Modelscited by 21
- 2022Carbon Footprint of Selecting and Training Deep Learning Models for Medical Image Analysiscited by 18
- 2020Towards Accurate and Reliable Energy Measurement of NLP Modelscited by 16
- 2021Winds of Change: Impact of COVID-19 on Vaccine-related Opinions of Twitter userscited by 16
- 2022CAVES: A Dataset to facilitate Explainable Classification and Summarization of Concerns towards COVID Vaccinescited by 16
- 2024Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inferencecited by 13
- 2021Towards Quantifying the Carbon Emissions of Differentially Private Machine Learningcited by 8
Built on by
Citing papers per year: 2025: 4; 2026: 7.
11 papers.
- 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
- 2025OckBench: Measuring the Efficiency of LLM Reasoningcited by 11
- 2026DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoningcited by 3
- 2025Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigationcited by 2
- 2026The Energy Footprint of LLM-Based Environmental Analysis: LLMs and Domain Productscited by 2
- 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platformscited by 2
- 2025LightCode: Compiling LLM Inference for Photonic-Electronic Systemscited by 0
- 2026The Global Landscape of Environmental AI Regulation: From the Cost of Reasoning to a Right to Green AIcited by 0
- 2026From Perception to Action: Can UI Interventions Foster Sustainable LLM Chatbotcited by 0
- 2026Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardwarecited by 0
- 2026Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architecturescited by 0
- 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms28 Aug 2026
- 2026Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures25 Aug 2026
- 2026Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware12 Jun 2026
- 2026From Perception to Action: Can UI Interventions Foster Sustainable LLM Chatbot9 Jun 2026
- 2026The Energy Footprint of LLM-Based Environmental Analysis: LLMs and Domain Products31 Mar 2026
- 2026The Global Landscape of Environmental AI Regulation: From the Cost of Reasoning to a Right to Green AI10 Feb 2026
- 2026DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoning28 Jan 2026
- 2025OckBench: Measuring the Efficiency of LLM Reasoning7 Nov 2025
- 2025Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigation22 Oct 2025
- 2025LightCode: Compiling LLM Inference for Photonic-Electronic Systems19 Sep 2025
- 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives27 May 2025
Similar, not cited
Close in wording, no citation either way
- 2025TokenPowerBench: Benchmarking the Power Consumption of LLM Inference
- 2024A Survey on Efficient Inference for Large Language Models
- 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
- 2024Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations
- 2024LLM Inference Unveiled: Survey and Roofline Model Insights
- 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
- 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute
- 2025Bench360: Benchmarking Local LLM Inference from 360 Degrees