# Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle

arXiv: https://arxiv.org/abs/2303.01980 · HTML: https://mlsanity.com/paper/2303.01980/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 85 of its 106 references that are in the map.
- First posted on arXiv 5 Feb 2023.

## Builds on (most cited first, 30 of 85)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)

## Built on by (most cited first, 14 of 14)

- [ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review](https://mlsanity.com/paper/2305.03123/) (2023)
- [ChatGPT in the Age of Generative AI and Large Language Models: A Concise Survey](https://mlsanity.com/paper/2307.04251/) (2023)
- [Neural Brain: A Neuroscience-inspired Framework for Embodied Agents](https://mlsanity.com/paper/2505.07634/) (2025)
- [On the Query Strategies for Efficient Online Active Distillation](https://mlsanity.com/paper/2309.01612/) (2023)
- [Investigation of Energy-efficient AI Model Architectures and Compression Techniques for "Green" Fetal Brain Segmentation](https://mlsanity.com/paper/2405.15778/) (2024)
- [Energy Consumption in Parallel Neural Network Training](https://mlsanity.com/paper/2508.07706/) (2025)
- [Cost-Aware Model Orchestration for LLM-based Systems](https://mlsanity.com/paper/2512.01099/) (2025)
- [Leveraging Hybrid Intelligence Towards Sustainable and Energy-Efficient Machine Learning](https://mlsanity.com/paper/2407.10580/) (2024)
- [From Theory to Practice: Implementing and Evaluating e-Fold Cross-Validation](https://mlsanity.com/paper/2410.09463/) (2024)
- [Phoeni6: a Systematic Approach for Evaluating the Energy Consumption of Neural Networks](https://mlsanity.com/paper/2502.17734/) (2025)
- [The HCI GenAI CO2ST Calculator: A Tool for Calculating the Carbon Footprint of Generative AI Use in Human-Computer Interaction Research](https://mlsanity.com/paper/2504.00692/) (2025)
- [Word Embedding Techniques for Classification of Star Ratings](https://mlsanity.com/paper/2504.13653/) (2025)
- [The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model](https://mlsanity.com/paper/2606.23546/) (2026)
- [Adaptive Computation with Conformal Certification](https://mlsanity.com/paper/2610.19189/) (2026)

## Abstract (arXiv)

Deep Learning has enabled many advances in machine learning applications in the last few years. However, since current Deep Learning algorithms require much energy for computations, there are growing concerns about the associated environmental costs. Energy-efficient Deep Learning has received much attention from researchers and has already made much progress in the last couple of years. This paper aims to gather information about these advances from the literature and show how and at which points along the lifecycle of Deep Learning (IT-Infrastructure, Data, Modeling, Training, Deployment, Evaluation) it is possible to reduce energy consumption.