# Algorithmic progress in language models

arXiv: https://arxiv.org/abs/2403.05812 · HTML: https://mlsanity.com/paper/2403.05812/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 9 Mar 2024.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?](https://mlsanity.com/paper/2207.10551/) (2022)
- [Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval](https://mlsanity.com/paper/2201.12431/) (2022)
- [Measuring the Algorithmic Efficiency of Neural Networks](https://mlsanity.com/paper/2005.04305/) (2020)
- [Language Models with Transformers](https://mlsanity.com/paper/1904.09408/) (2019)
- [AI capabilities can be significantly improved without expensive retraining](https://mlsanity.com/paper/2312.07413/) (2023)
- [Scaling Laws for Acoustic Models](https://mlsanity.com/paper/2106.09488/) (2021)
- [Algorithmic progress in computer vision](https://mlsanity.com/paper/2212.05153/) (2022)

## Built on by (most cited first, 30 of 31)

- [A Rosetta Stone for AI Benchmarks](https://mlsanity.com/paper/2512.00193/) (2025)
- [Position: The Future of Bayesian Prediction Is Prior-Fitted](https://mlsanity.com/paper/2505.23947/) (2025)
- [IDs for AI Systems](https://mlsanity.com/paper/2406.12137/) (2024)
- [On the Limitations of Compute Thresholds as a Governance Strategy](https://mlsanity.com/paper/2407.05694/) (2024)
- [SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers](https://mlsanity.com/paper/2609.01343/) (2026)
- [Training Compute Thresholds: Features and Functions in AI Regulation](https://mlsanity.com/paper/2405.10799/) (2024)
- [Learning to Plan for Language Modeling from Unlabeled Data](https://mlsanity.com/paper/2404.00614/) (2024)
- [AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement](https://mlsanity.com/paper/2608.20318/) (2026)
- [On the Origin of Algorithmic Progress in AI](https://mlsanity.com/paper/2511.21622/) (2025)
- [A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search](https://mlsanity.com/paper/2507.00004/) (2025)
- [Simulating Students with Large Language Models: A Review of Architecture, Mechanisms, and Role Modelling in Education with Generative AI](https://mlsanity.com/paper/2511.06078/) (2025)
- [Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems](https://mlsanity.com/paper/2503.01470/) (2025)
- [The Graph's Apprentice: Teaching an LLM Low Level Knowledge for Circuit Quality Estimation](https://mlsanity.com/paper/2411.00843/) (2024)
- [Note on Selection Bias in Observational Estimates of Algorithmic Progress](https://mlsanity.com/paper/2508.11033/) (2025)
- [The Price of Progress: Price Performance and the Future of AI](https://mlsanity.com/paper/2511.23455/) (2025)
- [Is there "Secret Sauce'' in Large Language Model Development?](https://mlsanity.com/paper/2602.07238/) (2026)
- [Rethinking LLM Advancement: Compute-Dependent and Independent Paths to Progress](https://mlsanity.com/paper/2505.04075/) (2025)
- [DICE: Data Influence Cascade in Decentralized Learning](https://mlsanity.com/paper/2507.06931/) (2025)
- [Technical Requirements for Halting Dangerous AI Activities](https://mlsanity.com/paper/2507.09801/) (2025)
- [Large Language Models, and LLM-Based Agents, Should Be Used to Enhance the Digital Public Sphere](https://mlsanity.com/paper/2410.12123/) (2024)
- [Reasoning about Affordances: Causal and Compositional Reasoning in LLMs](https://mlsanity.com/paper/2502.16606/) (2025)
- [Beyond Tokens: Enhancing RTL Quality Estimation via Structural Graph Learning](https://mlsanity.com/paper/2508.18730/) (2025)
- [Semantic Chunking and the Entropy of Natural Language](https://mlsanity.com/paper/2602.13194/) (2026)
- [AI and the Dynamic Supply of Training Data](https://mlsanity.com/paper/2404.18445/) (2024)
- [Will Neural Scaling Laws Activate Jevons' Paradox in AI Labor Markets? A Time-Varying Elasticity of Substitution (VES) Analysis](https://mlsanity.com/paper/2503.05816/) (2025)
- [Punctuated Equilibria in Artificial Intelligence: The Institutional Scaling Law and the Speciation of Sovereign AI](https://mlsanity.com/paper/2603.14664/) (2026)
- [Is the future of AI green? What can innovation diffusion models say about generative AI's environmental impact?](https://mlsanity.com/paper/2603.21419/) (2026)
- [Validity Threats for Foundation Model Research](https://mlsanity.com/paper/2606.05029/) (2026)
- [Sovereign by necessity? Frontier AI export controls, cyber security, and the limits of national AI capability](https://mlsanity.com/paper/2608.13272/) (2026)
- [Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence](https://mlsanity.com/paper/2608.14903/) (2026)

## Abstract (arXiv)

We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 95% confidence interval of around 5 to 14 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms.