mlsanityData to 14 Oct 2026

A Survey on Green Deep Learning

arXiv 2111.05193 ↗First posted 8 Nov 2021cs.LG cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

In recent years, larger and deeper models are springing up and continuously pushing state-of-the-art (SOTA) results across various fields like natural language processing (NLP) and computer vision (CV). However, despite promising results, it needs to be noted that the computations required by SOTA models have been increased at an exponential rate. Massive computations not only have a surprisingly large carbon footprint but also have negative effects on research inclusiveness and deployment on real-world applications. Green deep learning is an increasingly hot research field that appeals to researchers to pay attention to energy usage and carbon emission during model training and inference. The target is to yield novel results with lightweight and efficient technologies. Many technologies can be used to achieve this goal, like model compression and knowledge distillation. This paper focuses on presenting a systematic review of the development of Green deep learning technologies. We classify these approaches into four categories: (1) compact networks, (2) energy-efficient training strategies, (3) energy-efficient inference approaches, and (4) efficient data usage. For each category, we discuss the progress that has been achieved and the unresolved challenges.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 295 of its 393 references that are in the map. First posted on arXiv 8 Nov 2021.

Builds on

295 of 393 references.

References · 295

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
38 more
  1. 2017Mask R-CNNcited by 7,277
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  10. 2014Going Deeper with Convolutionscited by 4,241
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  13. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  14. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  15. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  16. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  17. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  18. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  19. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  20. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  21. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  22. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  23. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  24. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  25. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  26. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  27. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  28. 2018Deep contextualized word representationscited by 2,290
  29. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  30. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  31. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  32. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  33. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  34. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  35. 2019Searching for MobileNetV3cited by 1,994
  36. 2019Language Models as Knowledge Bases?cited by 1,936
  37. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  38. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818

50 most cited of 295 All 295 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 15; 2023: 19; 2024: 8; 2025: 4; 2026: 3.

52 papers.

Sort:

By area:

  1. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  2. 2022Confident Adaptive Language Modelingcited by 310
  3. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  4. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  5. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  6. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  7. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 90
  8. 2023A Systematic Review of Green AIcited by 77
  9. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  10. 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
  11. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  12. 2023Evaluating the Reversal Curse in Model Editingcited by 34
38 more
  1. 2023Interactive Natural Language Processingcited by 31
  2. 2022Green Learning: Introduction, Examples and Outlookcited by 30
  3. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  4. 2023Green Edge AI: A Contemporary Surveycited by 25
  5. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  6. 2021Distilled Dual-Encoder Model for Vision-Language Understandingcited by 22
  7. 2022Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learningcited by 22
  8. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  9. 2023Efficient Prompting via Dynamic In-Context Learningcited by 18
  10. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learnerscited by 17
  11. 2024A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trendscited by 16
  12. 2022MetAug: Contrastive Learning via Meta Feature Augmentationcited by 15
  13. 2023Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecyclecited by 14
  14. 2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
  15. 2021The Need for Ethical, Responsible, and Trustworthy Artificial Intelligence for Environmental Sciencescited by 12
  16. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Modelscited by 11
  17. 2023Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decodingcited by 10
  18. 2023Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Taskscited by 9
  19. 2023Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adaptercited by 8
  20. 2022A Primer on Topological Data Analysis to Support Image Analysis Tasks in Environmental Sciencecited by 7
  21. 2023SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Modelscited by 7
  22. 2023Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Searchcited by 5
  23. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inferencecited by 4
  24. 2022Distributional Correlation--Aware Knowledge Distillation for Stock Trading Volume Predictioncited by 2
  25. 2024Aggregate Representation Measure for Predictive Model Reusabilitycited by 2
  26. 2024Estimating Environmental Cost Throughout Model's Adaptive Life Cyclecited by 2
  27. 2023On the Query Strategies for Efficient Online Active Distillationcited by 1
  28. 2024An In-Depth Analysis of Data Reduction Methods for Sustainable Deep Learningcited by 1
  29. 2024Investigation of Energy-efficient AI Model Architectures and Compression Techniques for "Green" Fetal Brain Segmentationcited by 1
  30. 2024Neumann Series-based Neural Operator for Solving Inverse Medium Problemcited by 1
  31. 2025Where and How to Enhance: Discovering Bit-Width Contribution for Mixed Precision Quantizationcited by 1
  32. 2025A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?cited by 1
  33. 2026Active Learning Using Aggregated Acquisition Functions: Accuracy and Sustainability Analysiscited by 1
  34. 2023Structural Self-Supervised Objectives for Transformerscited by 0
  35. 2023GreenFlow: A Computation Allocation Framework for Building Environmentally Sound Recommendation Systemcited by 0
  36. 2024ssProp: Energy-Efficient Training for Convolutional Neural Networks with Scheduled Sparse Back Propagationcited by 0
  37. 2025Human Machine Co-Adaptation Model and Its Convergence Analysiscited by 0
  38. 2025Energy-Efficient Vision Transformer Inference for Edge-AI Deploymentcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024TinyM2Net-V3: Memory-Aware Compressed Multimodal Deep Neural Networks for Sustainable Edge Deployment
  2. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models
  3. 2025Optimizing Large Language Models: Metrics, Energy Efficiency, and Case Study Insights
  4. 2022The Ecological Footprint of Neural Machine Translation Systems
  5. 2025Deep Learning Innovations for Energy Efficiency: Advances in Non-Intrusive Load Monitoring and EV Charging Optimization for a Sustainable Grid
  6. 2024Latent Pollution Model: The Hidden Carbon Footprint in 3D Image Synthesis
  7. 2024A Decade of Deep Learning: A Survey on The Magnificent Seven
  8. 2025AOLO: Analysis and Optimization For Low-Carbon Oriented Wireless Large Language Model Services