# From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolution

arXiv: https://arxiv.org/abs/2404.06647 · HTML: https://mlsanity.com/paper/2404.06647/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 85 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research](https://mlsanity.com/paper/2112.01716/) (2021)
- [Impossibility Theorems for Feature Attribution](https://mlsanity.com/paper/2212.11870/) (2022)
- [Tradition and Innovation in Scientists' Research Strategies](https://mlsanity.com/paper/1302.6906/) (2013)

## Built on by (most cited first, 12 of 12)

- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culture](https://mlsanity.com/paper/2502.15620/) (2025)
- [What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text](https://mlsanity.com/paper/2503.04804/) (2025)
- [Computational Hermeneutics: Evaluating generative AI as a cultural technology](https://mlsanity.com/paper/2604.16403/) (2026)
- [Poor Alignment and Steerability of Large Language Models: Evidence from College Admission Essays](https://mlsanity.com/paper/2503.20062/) (2025)
- [Exploiting Leaderboards for Large-Scale Distribution of Malicious Models](https://mlsanity.com/paper/2507.08983/) (2025)
- [A Common Measure of Communication for Speech Brain-Computer Interfaces](https://mlsanity.com/paper/2609.02887/) (2026)
- [AI for Just Work: Constructing Diverse Imaginations of AI beyond "Replacing Humans"](https://mlsanity.com/paper/2503.08720/) (2025)
- [Reward Models are Metrics in a Trench Coat](https://mlsanity.com/paper/2510.03231/) (2025)
- [Text-to-Image Models Leave Identifiable Signatures: Implications for Leaderboard Security](https://mlsanity.com/paper/2510.06525/) (2025)
- [Knowing Your Uncertainty -- On the application of LLM in social sciences](https://mlsanity.com/paper/2512.05461/) (2025)
- [Position: Ideas Should be the Center of Machine Learning Research](https://mlsanity.com/paper/2605.15253/) (2026)

## Abstract (arXiv)

Over the past decade, AI research has focused heavily on building ever-larger deep learning models. This approach has simultaneously unlocked incredible achievements in science and technology, and hindered AI from overcoming long-standing limitations with respect to explainability, ethical harms, and environmental efficiency. Drawing on qualitative interviews and computational analyses, our three-part history of AI research traces the creation of this "epistemic monoculture" back to a radical reconceptualization of scientific progress that began in the late 1980s. In the first era of AI research (1950s-late 1980s), researchers and patrons approached AI as a "basic" science that would advance through autonomous exploration and organic assessments of progress (e.g., peer-review, theoretical consensus). The failure of this approach led to a retrenchment of funding in the 1980s. Amid this "AI Winter," an intervention by the U.S. government reoriented the field towards measurable progress on tasks of military and commercial interest. A new evaluation system called "benchmarking" provided an objective way to quantify progress on tasks by focusing exclusively on increasing predictive accuracy on example datasets. Distilling science down to verifiable metrics clarified the roles of scientists, allowed the field to rapidly integrate talent, and provided clear signals of significance and progress. But history has also revealed a tradeoff to this streamlined approach to science: the consolidation around external interests and inherent conservatism of benchmarking has disincentivized exploration beyond scaling monoculture. In the discussion, we explain how AI's monoculture offers a compelling challenge to the belief that basic, exploration-driven research is needed for scientific progress. Implications for the spread of AI monoculture to other sciences in the era of generative AI are also discussed.