# Note on Selection Bias in Observational Estimates of Algorithmic Progress

arXiv: https://arxiv.org/abs/2508.11033 · HTML: https://mlsanity.com/paper/2508.11033/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 3 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Builds on (most cited first, 3 of 3)

- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Chinchilla Scaling: A replication attempt](https://mlsanity.com/paper/2404.10102/) (2024)
- [Algorithmic progress in language models](https://mlsanity.com/paper/2403.05812/) (2024)

## Built on by (most cited first, 3 of 3)

- [Measuring AI R&D Automation](https://mlsanity.com/paper/2603.03992/) (2026)
- [On the Origin of Algorithmic Progress in AI](https://mlsanity.com/paper/2511.21622/) (2025)
- [Validity Threats for Foundation Model Research](https://mlsanity.com/paper/2606.05029/) (2026)

## Abstract (arXiv)

Ho et. al (2024) attempts to estimate the degree of algorithmic progress from language models. They collect observational data on language models' loss and compute over time, and argue that as time has passed, language models' algorithmic efficiency has been rising. That is, the loss achieved for fixed compute has been dropping over time. In this note, I raise one potential methodological problem with the estimation strategy. Intuitively, if part of algorithmic quality is latent, and compute choices are endogenous to algorithmic quality, then resulting estimates of algorithmic quality will be contaminated by selection bias.