mlsanityData to 7 Oct 2026

The Computational Limits of Deep Learning

arXiv 2007.05558 ↗First posted 10 Jul 2020cs.LG stat.MLcited by 188

Not on the map. Why?

Abstract (arXiv)

Deep learning's recent history has been one of achievement: from triumphing over humans in the game of Go to world-leading performance in image classification, voice recognition, translation, and other tasks. But this progress has come with a voracious appetite for computing power. This article catalogs the extent of this dependency, showing that progress across a wide variety of applications is strongly reliant on increases in computing power. Extrapolating forward this reliance reveals that progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable. Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods.

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 35; 2023: 40; 2024: 39; 2025: 27; 2026: 20.

188 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  4. 2021Training Spiking Neural Networks Using Lessons From Deep Learningcited by 251
  5. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 183
  6. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  7. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  8. 2022Measuring the Carbon Intensity of AI in Cloud Instancescited by 110
  9. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  10. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  11. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  12. 2021Compute and Energy Consumption Trends in Deep Learning Inferencecited by 61
38 more
  1. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  2. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  3. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
  4. 2023Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learningcited by 42
  5. 2021Incremental Learning in Semantic Segmentation from Image Labelscited by 39
  6. 2022Recent Advances for Quantum Neural Networks in Generative Learningcited by 39
  7. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  8. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  9. 2022Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computingcited by 34
  10. 2023How to estimate carbon footprint when training deep learning models? A guide and reviewcited by 34
  11. 2021An optical neural network using less than 1 photon per multiplicationcited by 30
  12. 2022On the influence of over-parameterization in manifold based surrogates and deep neural operatorscited by 29
  13. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
  14. 2023A Machine Learning-oriented Survey on Tiny Machine Learningcited by 26
  15. 2024Telco-RAG: Navigating the Challenges of Retrieval-Augmented Language Models for Telecommunicationscited by 26
  16. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  17. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  18. 2021Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligencecited by 21
  19. 2022Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Modelscited by 21
  20. 2023Not all layers are equally as important: Every Layer Counts BERTcited by 21
  21. 2025Continuous Thought Machinescited by 21
  22. 2021SplitGuard: Detecting and Mitigating Training-Hijacking Attacks in Split Learningcited by 20
  23. 2023To Be Forgotten or To Be Fair: Unveiling Fairness Implications of Machine Unlearning Methodscited by 18
  24. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  25. 2021Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networkscited by 15
  26. 2023Holistic Network Virtualization and Pervasive Network Intelligence for 6Gcited by 14
  27. 2022Algorithmic progress in computer visioncited by 13
  28. 2021Energy Consumption of Deep Generative Audio Modelscited by 12
  29. 2023Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AIcited by 12
  30. 2022A Hybrid Quantum-Classical Neural Network Architecture for Binary Classificationcited by 11
  31. 2022Metaethical Perspectives on 'Benchmarking' AI Ethicscited by 11
  32. 2023Accuracy is not the only Metric that matters: Estimating the Energy Consumption of Deep Learning Modelscited by 9
  33. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  34. 2023Multiplexed gradient descent: Fast online training of modern datasets on hardware neural networks without backpropagationcited by 8
  35. 2023Photonic Advantage of Optical Encoderscited by 8
  36. 2024ICE-SEARCH: A Language Model-Driven Feature Selection Approachcited by 8
  37. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  38. 2021Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networkscited by 7

50 most cited of 188 All 188 (JSON)