mlsanityData to 7 Oct 2026

Carbon Emissions and Large Neural Network Training

arXiv 2104.10350 ↗First posted 21 Apr 2021cs.LG cs.CYcited by 561

Not on the map. Why?

Abstract (arXiv)

The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs. Estimating the energy cost helps measure its environmental impact and finding greener strategies, yet it is challenging without detailed information. We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3-and refine earlier estimates for the neural architecture search that found Evolved Transformer. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e): Large but sparsely activated DNNs can consume <1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters. Geographic location matters for ML workload scheduling since the fraction of carbon-free energy and resulting CO2e vary ~5X-10X, even within the same country and the same organization. We are now optimizing where and when large models are trained. Specific datacenter infrastructure matters, as Cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators inside them can be ~2-5X more effective than off-the-shelf systems. Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X. These large factors also make retroactive estimates of energy cost difficult. To avoid miscalculations, we believe ML papers requiring large computational resources should make energy consumption and CO2e explicit when practical. We are working to be more transparent about energy use and CO2e in our future research. To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models, and we are collaborating with MLPerf developers to include energy usage during training and inference in this industry standard benchmark.

Key facts. Cited by 561 papers in the map (data through 7 Oct 2026). Builds on 27 of its 59 references that are in the map. First posted on arXiv 21 Apr 2021.

Builds on

27 of 59 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,782
15 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  3. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  4. 2019Green AIcited by 620
  5. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  6. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  7. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 462
  8. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  9. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  10. 2019The Evolved Transformercited by 180
  11. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  12. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 150
  13. 2020Chasing Carbon: The Elusive Environmental Footprint of Computingcited by 71
  14. 2020Measuring the Algorithmic Efficiency of Neural Networkscited by 51
  15. 2021Searching for Fast Model Families on Datacenter Acceleratorscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 77; 2023: 98; 2024: 99; 2025: 121; 2026: 115.

561 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Segment Anythingcited by 8,158
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2022Mass-Editing Memory in a Transformercited by 976
38 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 957
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  3. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  4. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  7. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  8. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  9. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  10. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  11. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  12. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  13. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  14. 2022Metrics reloaded: Recommendations for image analysis validationcited by 190
  15. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  16. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 186
  17. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  18. 2022Parameter-free Online Test-time Adaptationcited by 184
  19. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 183
  20. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  21. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  22. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  23. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  24. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  25. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  26. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  27. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  28. 2023Large language models can accurately predict searcher preferencescited by 129
  29. 2021Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systemscited by 125
  30. 2023Factuality Challenges in the Era of Large Language Modelscited by 125
  31. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  32. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  33. 2023GEO-Bench: Toward Foundation Models for Earth Monitoringcited by 122
  34. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  35. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  36. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  37. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  38. 2022Measuring the Carbon Intensity of AI in Cloud Instancescited by 110

50 most cited of 561 All 561 (JSON)