mlsanityData to 7 Oct 2026

Measuring the Algorithmic Efficiency of Neural Networks

arXiv 2005.04305 ↗First posted 8 May 2020cs.LG cs.CV stat.MLcited by 51

Not on the map. Why?

Abstract (arXiv)

Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to quantify than compute and data. In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmic efficiency doubling every 16 months over a period of 7 years. By contrast, Moore's Law would only have yielded an 11x cost improvement. We observe that hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress should integrate measures from both.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 15; 2022: 9; 2023: 7; 2024: 6; 2025: 6; 2026: 7.

51 papers.

Sort:

By area:

  1. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  2. 2021Scaling Laws for Transfercited by 265
  3. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  4. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  5. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  6. 2021Compute and Energy Consumption Trends in Deep Learning Inferencecited by 61
  7. 2025Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimizationcited by 52
  8. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  9. 2024Algorithmic progress in language modelscited by 31
  10. 2023AI capabilities can be significantly improved without expensive retrainingcited by 28
  11. 2022Benchmarking simulated and physical quantum processing units using quantum and hybrid algorithmscited by 18
  12. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
38 more
  1. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  2. 2021Automated Essay Scoring Using Transformer Modelscited by 14
  3. 2021Joint Superposition Coding and Training for Federated Learning over Multi-Width Neural Networkscited by 14
  4. 2022Algorithmic progress in computer visioncited by 13
  5. 2024Training Compute Thresholds: Features and Functions in AI Regulationcited by 13
  6. 2021Energy Consumption of Deep Generative Audio Modelscited by 12
  7. 2021SECDA: Efficient Hardware/Software Co-Design of FPGA-based DNN Accelerators for Edge Inferencecited by 12
  8. 2023Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AIcited by 12
  9. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 12
  10. 2025Life-Cycle Emissions of AI Hardware: A Cradle-To-Grave Approach and Generational Trendscited by 10
  11. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  12. 2021Greenformer: Factorization Toolkit for Efficient Deep Neural Networkscited by 7
  13. 2025PredictaBoard: Benchmarking LLM Score Predictabilitycited by 7
  14. 2026Measuring AI R&D Automationcited by 7
  15. 2022SlimFL: Federated Learning with Superposition Coding over Slimmable Neural Networkscited by 6
  16. 2021Deep Reinforcement Learning for Conservation Decisionscited by 5
  17. 2021Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximationcited by 5
  18. 2024Learn from Foundation Model: Fruit Detection Model without Manual Annotationcited by 5
  19. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  20. 2025On the Origin of Algorithmic Progress in AIcited by 5
  21. 2021Accelerating Federated Learning with a Global Biased Optimisercited by 4
  22. 2022Position: Tensor Networks are a Valuable Asset for Green AIcited by 3
  23. 2023Criticality versus uniformity in deep neural networkscited by 3
  24. 2024The Graph's Apprentice: Teaching an LLM Low Level Knowledge for Circuit Quality Estimationcited by 3
  25. 2021Efficient and Generic 1D Dilated Convolution Layer for Deep Learningcited by 1
  26. 2022Bifrost: End-to-End Evaluation and Optimization of Reconfigurable DNN Acceleratorscited by 1
  27. 2022Capturing Dependencies within Machine Learning via a Formal Process Modelcited by 1
  28. 2022Towards Sustainable Artificial Intelligence: An Overview of Environmental Protection Uses and Issuescited by 1
  29. 2021Distributed Optimization using Heterogeneous Compute Systemscited by 0
  30. 2023On Inductive Biases for Machine Learning in Data Constrained Settingscited by 0
  31. 2023Neural network scoring for efficient computingcited by 0
  32. 2023DLAS: An Exploration and Assessment of the Deep Learning Acceleration Stackcited by 0
  33. 2024A Framework to Enable Algorithmic Design Choice Exploration in DNNscited by 0
  34. 2025Complexity as Advantage: A Regret-Based Perspective on Emergent Structurecited by 0
  35. 2026Continued AI Scaling Requires Repeated Efficiency Doublingscited by 0
  36. 2026Scalable Reinforcement Learning via Adaptive Batch Scalingcited by 0
  37. 2026Have I Solved This Before? Retrieving Similar Segmentation Problems for Evolutionary Learningcited by 0
  38. 2026Grounded Scaling: Why Agentic AI Needs Deterministic Environmentscited by 0

50 most cited of 51 All 51 (JSON)