# Measuring the Algorithmic Efficiency of Neural Networks

arXiv: https://arxiv.org/abs/2005.04305 · HTML: https://mlsanity.com/paper/2005.04305/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2020.

## Built on by (most cited first, 30 of 52)

- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [Scaling Laws and Interpretability of Learning from Repeated Data](https://mlsanity.com/paper/2205.10487/) (2022)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Compute and Energy Consumption Trends in Deep Learning Inference](https://mlsanity.com/paper/2109.05472/) (2021)
- [Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization](https://mlsanity.com/paper/2509.14279/) (2025)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Algorithmic progress in language models](https://mlsanity.com/paper/2403.05812/) (2024)
- [AI capabilities can be significantly improved without expensive retraining](https://mlsanity.com/paper/2312.07413/) (2023)
- [Benchmarking simulated and physical quantum processing units using quantum and hybrid algorithms](https://mlsanity.com/paper/2211.15631/) (2022)
- [On the Limitations of Compute Thresholds as a Governance Strategy](https://mlsanity.com/paper/2407.05694/) (2024)
- [SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers](https://mlsanity.com/paper/2609.01343/) (2026)
- [Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.16785/) (2020)
- [Automated Essay Scoring Using Transformer Models](https://mlsanity.com/paper/2110.06874/) (2021)
- [Joint Superposition Coding and Training for Federated Learning over Multi-Width Neural Networks](https://mlsanity.com/paper/2112.02543/) (2021)
- [Algorithmic progress in computer vision](https://mlsanity.com/paper/2212.05153/) (2022)
- [Training Compute Thresholds: Features and Functions in AI Regulation](https://mlsanity.com/paper/2405.10799/) (2024)
- [Energy Consumption of Deep Generative Audio Models](https://mlsanity.com/paper/2107.02621/) (2021)
- [SECDA: Efficient Hardware/Software Co-Design of FPGA-based DNN Accelerators for Edge Inference](https://mlsanity.com/paper/2110.00478/) (2021)
- [Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AI](https://mlsanity.com/paper/2309.02065/) (2023)
- [Life-Cycle Emissions of AI Hardware: A Cradle-To-Grave Approach and Generational Trends](https://mlsanity.com/paper/2502.01671/) (2025)
- [Measuring AI R&D Automation](https://mlsanity.com/paper/2603.03992/) (2026)
- [Measuring Progress in Deep Reinforcement Learning Sample Efficiency](https://mlsanity.com/paper/2102.04881/) (2021)
- [Greenformer: Factorization Toolkit for Efficient Deep Neural Networks](https://mlsanity.com/paper/2109.06762/) (2021)
- [PredictaBoard: Benchmarking LLM Score Predictability](https://mlsanity.com/paper/2502.14445/) (2025)
- [SlimFL: Federated Learning with Superposition Coding over Slimmable Neural Networks](https://mlsanity.com/paper/2203.14094/) (2022)
- [On the Origin of Algorithmic Progress in AI](https://mlsanity.com/paper/2511.21622/) (2025)
- [Deep Reinforcement Learning for Conservation Decisions](https://mlsanity.com/paper/2106.08272/) (2021)
- [Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation](https://mlsanity.com/paper/2108.10808/) (2021)

## Abstract (arXiv)

Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to quantify than compute and data. In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmic efficiency doubling every 16 months over a period of 7 years. By contrast, Moore's Law would only have yielded an 11x cost improvement. We observe that hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress should integrate measures from both.