mlsanityData to 7 Oct 2026

Energy and Policy Considerations for Deep Learning in NLP

arXiv 1906.02243 ↗First posted 5 Jun 2019cs.CLcited by 1,242

Not on the map. Why?

Abstract (arXiv)

Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.

Key facts. Cited by 1,242 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 246; 2022: 243; 2023: 235; 2024: 174; 2025: 151; 2026: 193.

1,242 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2021Ethical and social risks of harm from Language Modelscited by 957
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  3. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  4. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  5. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  6. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  7. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  8. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  9. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  10. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  11. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  12. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  13. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  14. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  15. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  16. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  17. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  18. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  19. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  20. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  21. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  22. 2022Metrics reloaded: Recommendations for image analysis validationcited by 190
  23. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 186
  24. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  25. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 183
  26. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  27. 2021Challenges in Detoxifying Language Modelscited by 169
  28. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  29. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  30. 2023A Categorical Archive of ChatGPT Failurescited by 167
  31. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  32. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  33. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
  34. 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 145
  35. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  36. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  37. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  38. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134

50 most cited of 1,242 All 1,242 (JSON)