Energy-Conscious LLM Decoding: Impact of Text Generation Strategies on GPU Energy Consumption
Not on the map. Why?
Abstract (arXiv)
Decoding strategies significantly influence the quality and diversity of the generated text in Large Language Models (LLMs), yet their impact on computational resources, particularly GPU energy consumption, is insufficiently studied. This paper investigates the relationship between text generation decoding techniques and energy efficiency, focusing on the trade-off between generation quality and GPU energy usage across diverse tasks and decoding configurations. By benchmarking multiple strategies across various tasks, including Translation, Math Problem Solving, Coding, and Open-ended text generation, we reveal how selecting appropriate decoding techniques with their tuned hyperparameters affects text quality and has measurable implications for energy consumption. Our findings show that the choice of decoding strategy can greatly impact GPU energy usage, even when it has a minimal effect on output quality. Different strategies also involve trade-offs between quality and energy efficiency, and no single decoding method is best in all cases across every metric. To the best of our knowledge, this is one of the first studies to examine decoding strategies in LLMs from the perspective of energy consumption, providing useful insights for building energy-efficient applications without compromising text generation quality.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 39 of its 60 references that are in the map. First posted on arXiv 17 Feb 2025.
Builds on
39 of 60 references.
References · 39
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2019The Curious Case of Neural Text Degenerationcited by 2,916
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
- 2018Hierarchical Neural Story Generationcited by 1,193
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
27 more
- 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
- 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
- 2021Carbon Emissions and Large Neural Network Trainingcited by 567
- 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
- 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
- 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
- 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
- 2017Beam Search Strategies for Neural Machine Translationcited by 232
- 2022A Contrastive Framework for Neural Text Generationcited by 225
- 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
- 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
- 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
- 2022Truncation Sampling as Language Model Desmoothingcited by 123
- 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 114
- 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
- 2022Locally Typical Samplingcited by 111
- 2022On Decoding Strategies for Neural Text Generatorscited by 90
- 2020Consistency of a Recurrent Language Model With Respect to Incomplete Decodingcited by 54
- 2024Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generationcited by 26
- 2022Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Modelscited by 21
- 2023Part-time Power Measurements: nvidia-smi's Lack of Attentioncited by 15
- 2022An Empirical Study On Contrastive Search And Contrastive Decoding For Open-ended Text Generationcited by 14
- 2023Toward Sustainable HPC: Carbon Footprint Estimation and Environmental Implications of HPC Systemscited by 12
- 2024Sustainable Supercomputing for AI: GPU Power Capping at HPC Scalecited by 9
- 2024Double-Exponential Increases in Inference Energy: The Cost of the Race for Accuracycited by 6
Built on by
Citing papers per year: 2025: 1.
1 paper.