mlsanityData to 7 Oct 2026

Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle

arXiv 2303.01980 ↗First posted 5 Feb 2023cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Deep Learning has enabled many advances in machine learning applications in the last few years. However, since current Deep Learning algorithms require much energy for computations, there are growing concerns about the associated environmental costs. Energy-efficient Deep Learning has received much attention from researchers and has already made much progress in the last couple of years. This paper aims to gather information about these advances from the literature and show how and at which points along the lifecycle of Deep Learning (IT-Infrastructure, Data, Modeling, Training, Deployment, Evaluation) it is possible to reduce energy consumption.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 85 of its 106 references that are in the map. First posted on arXiv 5 Feb 2023.

Builds on

85 of 106 references.

References · 85

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2016Layer Normalizationcited by 5,579
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
38 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2020Knowledge Distillation: A Surveycited by 1,578
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2018Group Normalizationcited by 1,514
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  8. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  9. 2017Mixed Precision Trainingcited by 1,135
  10. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  11. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  13. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  14. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  15. 2019Green AIcited by 620
  16. 2018Universal Transformerscited by 613
  17. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  18. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  19. 2020A Survey of Deep Active Learningcited by 547
  20. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  21. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  22. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  23. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  24. 2018Slimmable Neural Networkscited by 301
  25. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  26. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  27. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  28. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  29. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 168
  30. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  31. 2018Neural Architecture Optimizationcited by 164
  32. 2019Adaptively Sparse Transformerscited by 154
  33. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  34. 2017Convolutional Networks with Adaptive Inference Graphscited by 147
  35. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  36. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  37. 2017CondenseNet: An Efficient DenseNet using Learned Group Convolutionscited by 121
  38. 2019Neural Machine Translation with Byte-Level Subwordscited by 115

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 6; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Reviewcited by 28
  2. 2023ChatGPT in the Age of Generative AI and Large Language Models: A Concise Surveycited by 10
  3. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  4. 2023On the Query Strategies for Efficient Online Active Distillationcited by 1
  5. 2024Investigation of Energy-efficient AI Model Architectures and Compression Techniques for "Green" Fetal Brain Segmentationcited by 1
  6. 2025Energy Consumption in Parallel Neural Network Trainingcited by 1
  7. 2025Cost-Aware Model Orchestration for LLM-based Systemscited by 1
  8. 2024Leveraging Hybrid Intelligence Towards Sustainable and Energy-Efficient Machine Learningcited by 0
  9. 2024From Theory to Practice: Implementing and Evaluating e-Fold Cross-Validationcited by 0
  10. 2025Phoeni6: a Systematic Approach for Evaluating the Energy Consumption of Neural Networkscited by 0
  11. 2025The HCI GenAI CO2ST Calculator: A Tool for Calculating the Carbon Footprint of Generative AI Use in Human-Computer Interaction Researchcited by 0
  12. 2025Word Embedding Techniques for Classification of Star Ratingscited by 0
1 more
  1. 2026The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Modelcited by 0