# Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models

arXiv: https://arxiv.org/abs/2205.09646 · HTML: https://mlsanity.com/paper/2205.09646/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [The Computational Limits of Deep Learning](https://mlsanity.com/paper/2007.05558/) (2020)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [An Analysis of Deep Neural Network Models for Practical Applications](https://mlsanity.com/paper/1605.07678/) (2016)
- [Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis](https://mlsanity.com/paper/1807.07814/) (2018)
- [The MIT Supercloud Dataset](https://mlsanity.com/paper/2108.02037/) (2021)
- [Benchmarking Resource Usage for Efficient Distributed Deep Learning](https://mlsanity.com/paper/2201.12423/) (2022)

## Built on by (most cited first, 21 of 21)

- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2406.08413/) (2024)
- [Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models](https://mlsanity.com/paper/2502.05610/) (2025)
- [The Unseen AI Disruptions for Power Grids: LLM-Induced Transients](https://mlsanity.com/paper/2409.11416/) (2024)
- [Quantum Transformer: Accelerating model inference via quantum linear algebra](https://mlsanity.com/paper/2402.16714/) (2024)
- [Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale](https://mlsanity.com/paper/2402.18593/) (2024)
- [Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform](https://mlsanity.com/paper/2505.11892/) (2025)
- [The Power of Training: How Different Neural Network Setups Influence the Energy Demand](https://mlsanity.com/paper/2401.01851/) (2024)
- [Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse](https://mlsanity.com/paper/2505.16284/) (2025)
- [Mixture of Tokens: Continuous MoE through Cross-Example Aggregation](https://mlsanity.com/paper/2310.15961/) (2023)
- [Generating Energy-efficient code with LLMs](https://mlsanity.com/paper/2411.10599/) (2024)
- [HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis](https://mlsanity.com/paper/2405.00738/) (2024)
- [Estimating Environmental Cost Throughout Model's Adaptive Life Cycle](https://mlsanity.com/paper/2408.01446/) (2024)
- [Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems](https://mlsanity.com/paper/2410.23472/) (2024)
- [Energy-Conscious LLM Decoding: Impact of Text Generation Strategies on GPU Energy Consumption](https://mlsanity.com/paper/2502.11723/) (2025)
- [SONIQ: System-Optimized Noise-Injected Ultra-Low-Precision Quantization with Full-Precision Parity](https://mlsanity.com/paper/2311.14114/) (2023)
- [Adaptive Inference: Theoretical Limits and Unexplored Opportunities](https://mlsanity.com/paper/2402.04359/) (2024)
- [MetaGreen: Meta-Learning Inspired Transformer Selection for Green Semantic Communication](https://mlsanity.com/paper/2406.16962/) (2024)
- [RESQUE: Quantifying Estimator to Task and Distribution Shift for Sustainable Model Reusability](https://mlsanity.com/paper/2412.15511/) (2024)

## Abstract (arXiv)

The energy requirements of current natural language processing models continue to grow at a rapid, unsustainable pace. Recent works highlighting this problem conclude there is an urgent need for methods that reduce the energy needs of NLP and machine learning more broadly. In this article, we investigate techniques that can be used to reduce the energy consumption of common NLP applications. In particular, we focus on techniques to measure energy usage and different hardware and datacenter-oriented settings that can be tuned to reduce energy consumption for training and inference for language models. We characterize the impact of these settings on metrics such as computational performance and energy consumption through experiments conducted on a high performance computing system as well as popular cloud computing platforms. These techniques can lead to significant reduction in energy consumption when training language models or their use for inference. For example, power-capping, which limits the maximum power a GPU can consume, can enable a 15% decrease in energy usage with marginal increase in overall computation time when training a transformer-based language model.