# Watt For What: Rethinking Deep Learning's Energy-Performance Relationship

arXiv: https://arxiv.org/abs/2310.06522 · HTML: https://mlsanity.com/paper/2310.06522/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 78 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 30 of 62)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)

## Built on by (most cited first, 16 of 16)

- [PEM: Prototype-based Efficient MaskFormer for Image Segmentation](https://mlsanity.com/paper/2402.19422/) (2024)
- [Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Survey](https://mlsanity.com/paper/2505.01821/) (2025)
- [Principles of Visual Tokens for Efficient Video Understanding](https://mlsanity.com/paper/2411.13626/) (2024)
- [CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization](https://mlsanity.com/paper/2505.01386/) (2025)
- [FE-Adapter: Adapting Image-based Emotion Classifiers to Videos](https://mlsanity.com/paper/2408.02421/) (2024)
- [Progressive Data Dropout: An Embarrassingly Simple Approach to Faster Training](https://mlsanity.com/paper/2505.22342/) (2025)
- [Masks and Manuscripts: Advancing Medical Pre-training with End-to-End Masking and Narrative Structuring](https://mlsanity.com/paper/2407.16264/) (2024)
- [SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version](https://mlsanity.com/paper/2508.15478/) (2025)
- [Return of EM: Entity-driven Answer Set Expansion for QA Evaluation](https://mlsanity.com/paper/2404.15650/) (2024)
- [Green AI in Action: Strategic Model Selection for Ensembles in Production](https://mlsanity.com/paper/2405.17451/) (2024)
- [The Hidden Cost of an Image: Quantifying the Energy Consumption of AI Image Generation](https://mlsanity.com/paper/2506.17016/) (2025)
- [Local learning for stable backpropagation-free neural network training towards physical learning](https://mlsanity.com/paper/2603.24790/) (2026)
- [Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks](https://mlsanity.com/paper/2604.12945/) (2026)
- [Performance is not All You Need: Sustainability Considerations for Algorithms](https://mlsanity.com/paper/2509.00045/) (2025)
- [Energy-Efficient Vision Transformer Inference for Edge-AI Deployment](https://mlsanity.com/paper/2511.23166/) (2025)
- [Using Explainability as a Training-Time Reliability Signal for Efficient ECG Classification](https://mlsanity.com/paper/2606.12252/) (2026)

## Abstract (arXiv)

Deep learning models have revolutionized various fields, from image recognition to natural language processing, by achieving unprecedented levels of accuracy. However, their increasing energy consumption has raised concerns about their environmental impact, disadvantaging smaller entities in research and exacerbating global energy consumption. In this paper, we explore the trade-off between model accuracy and electricity consumption, proposing a metric that penalizes large consumption of electricity. We conduct a comprehensive study on the electricity consumption of various deep learning models across different GPUs, presenting a detailed analysis of their accuracy-efficiency trade-offs. By evaluating accuracy per unit of electricity consumed, we demonstrate how smaller, more energy-efficient models can significantly expedite research while mitigating environmental concerns. Our results highlight the potential for a more sustainable approach to deep learning, emphasizing the importance of optimizing models for efficiency. This research also contributes to a more equitable research landscape, where smaller entities can compete effectively with larger counterparts. This advocates for the adoption of efficient deep learning practices to reduce electricity consumption, safeguarding the environment for future generations whilst also helping ensure a fairer competitive landscape.