# The Computational Limits of Deep Learning

arXiv: https://arxiv.org/abs/2007.05558 · HTML: https://mlsanity.com/paper/2007.05558/

## Key facts

- Cited by 193 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2020.

## Built on by (most cited first, 30 of 193)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink](https://mlsanity.com/paper/2204.05149/) (2022)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [Avalanche: an End-to-End Library for Continual Learning](https://mlsanity.com/paper/2104.00405/) (2021)
- [Measuring the Carbon Intensity of AI in Cloud Instances](https://mlsanity.com/paper/2206.05229/) (2022)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity](https://mlsanity.com/paper/2301.12867/) (2023)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [Compute and Energy Consumption Trends in Deep Learning Inference](https://mlsanity.com/paper/2109.05472/) (2021)
- [FinTextQA: A Dataset for Long-form Financial Question Answering](https://mlsanity.com/paper/2405.09980/) (2024)
- [The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation](https://mlsanity.com/paper/2110.03036/) (2021)
- [Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models](https://mlsanity.com/paper/2404.10237/) (2024)
- [Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learning](https://mlsanity.com/paper/2302.08476/) (2023)
- [Incremental Learning in Semantic Segmentation from Image Labels](https://mlsanity.com/paper/2112.01882/) (2021)
- [Recent Advances for Quantum Neural Networks in Generative Learning](https://mlsanity.com/paper/2206.03066/) (2022)
- [Machine Learning Model Sizes and the Parameter Gap](https://mlsanity.com/paper/2207.02852/) (2022)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computing](https://mlsanity.com/paper/2210.15889/) (2022)
- [How to estimate carbon footprint when training deep learning models? A guide and review](https://mlsanity.com/paper/2306.08323/) (2023)
- [An optical neural network using less than 1 photon per multiplication](https://mlsanity.com/paper/2104.13467/) (2021)
- [On the influence of over-parameterization in manifold based surrogates and deep neural operators](https://mlsanity.com/paper/2203.05071/) (2022)
- [Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?](https://mlsanity.com/paper/2303.04143/) (2023)
- [A Machine Learning-oriented Survey on Tiny Machine Learning](https://mlsanity.com/paper/2309.11932/) (2023)
- [Telco-RAG: Navigating the Challenges of Retrieval-Augmented Language Models for Telecommunications](https://mlsanity.com/paper/2404.15939/) (2024)
- [Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization](https://mlsanity.com/paper/2102.01670/) (2021)
- [An Overview on Language Models: Recent Developments and Outlook](https://mlsanity.com/paper/2303.05759/) (2023)
- [Continuous Thought Machines](https://mlsanity.com/paper/2505.05522/) (2025)

## Abstract (arXiv)

Deep learning's recent history has been one of achievement: from triumphing over humans in the game of Go to world-leading performance in image classification, voice recognition, translation, and other tasks. But this progress has come with a voracious appetite for computing power. This article catalogs the extent of this dependency, showing that progress across a wide variety of applications is strongly reliant on increases in computing power. Extrapolating forward this reliance reveals that progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable. Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods.