mlsanityData to 7 Oct 2026

Compute Trends Across Three Eras of Machine Learning

arXiv 2202.05924 ↗First posted 11 Feb 2022cs.LG cs.AI cs.CYcited by 161

Not on the map. Why?

Abstract (arXiv)

Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore's law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute. Based on these observations we split the history of compute in ML into three eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Builds on 75 of its 181 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

75 of 181 references.

References · 75

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  8. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  9. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  10. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2016Layer Normalizationcited by 5,579
38 more
  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  2. 2014Going Deeper with Convolutionscited by 4,231
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  13. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  14. 2018YOLOv3: An Incremental Improvementcited by 2,652
  15. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  16. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  17. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  18. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  19. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  20. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  21. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  22. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  23. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  24. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  25. 2021Scaling Vision Transformerscited by 993
  26. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  27. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  28. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  29. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  30. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  31. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  32. 2014Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognitioncited by 719
  33. 2019Solving Rubik's Cube with a Robot Handcited by 699
  34. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  35. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  36. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  37. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  38. 2017Progressive Neural Architecture Searchcited by 483

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 42; 2024: 35; 2025: 31; 2026: 37.

161 papers.

Sort:

By area:

  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  2. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  3. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  4. 2025Measuring AI Ability to Complete Long Software Taskscited by 142
  5. 2023Auditing large language models: a three-layered approachcited by 135
  6. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  7. 2023EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Modelscited by 101
  8. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  9. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  10. 2023Eight Things to Know about Large Language Modelscited by 73
  11. 2023Spear Phishing With Large Language Modelscited by 49
  12. 2023Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based acceleratorscited by 42
38 more
  1. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  2. 2022Large Language Models and the Reverse Turing Testcited by 38
  3. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  4. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 37
  5. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  6. 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
  7. 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
  8. 2024Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initializationcited by 29
  9. 2023ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Reviewcited by 28
  10. 2023NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systemscited by 27
  11. 2023Green Edge AI: A Contemporary Surveycited by 24
  12. 2024SpiNNaker2: A Large-Scale Neuromorphic System for Event-Based and Asynchronous Machine Learningcited by 24
  13. 2023What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoringcited by 20
  14. 2024The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?cited by 20
  15. 2022Towards Climate Awareness in NLP Researchcited by 19
  16. 2023A Material Lens on Coloniality in NLPcited by 19
  17. 2022Carbon Footprint of Selecting and Training Deep Learning Models for Medical Image Analysiscited by 18
  18. 2022Astronomia ex machina: a history, primer, and outlook on neural networks in astronomycited by 17
  19. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  20. 2023How Generative AI models such as ChatGPT can be (Mis)Used in SPC Practice, Education, and Research? An Exploratory Studycited by 16
  21. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  22. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 16
  23. 2023THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compressioncited by 15
  24. 2023Market Concentration Implications of Foundation Modelscited by 15
  25. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  26. 2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 15
  27. 2024Scenarios for the Transition to AGIcited by 13
  28. 2024Training Compute Thresholds: Features and Functions in AI Regulationcited by 13
  29. 2025LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviewscited by 13
  30. 2023Spatially Varying Nanophotonic Neural Networkscited by 12
  31. 2023Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AIcited by 12
  32. 2023TRANSOM: An Efficient Fault-Tolerant System for Training LLMscited by 12
  33. 2022EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Searchcited by 11
  34. 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregationcited by 11
  35. 2022Federated Self-supervised Speech Representations: Are We There Yet?cited by 10
  36. 2023Optical Transformerscited by 10
  37. 2024The Unseen AI Disruptions for Power Grids: LLM-Induced Transientscited by 10
  38. 2026KernelBlaster: Continual Cross-Task CUDA Optimization via Memory-Augmented In-Context Reinforcement Learningcited by 10

50 most cited of 161 All 161 (JSON)