# The Scientific Method in the Science of Machine Learning

arXiv: https://arxiv.org/abs/1904.10922 · HTML: https://mlsanity.com/paper/1904.10922/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2019.

## Built on by (most cited first, 15 of 15)

- [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [Modeling the Machine Learning Multiverse](https://mlsanity.com/paper/2206.05985/) (2022)
- [STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports](https://mlsanity.com/paper/2508.09853/) (2025)
- [Perspectives on Machine Learning from Psychology's Reproducibility Crisis](https://mlsanity.com/paper/2104.08878/) (2021)
- [Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor](https://mlsanity.com/paper/2506.14652/) (2025)
- [Expected Validation Performance and Estimation of a Random Variable's Maximum](https://mlsanity.com/paper/2110.00613/) (2021)
- [Learning from learning machines: a new generation of AI technology to meet the needs of science](https://mlsanity.com/paper/2111.13786/) (2021)
- [Towards Inferential Reproducibility of Machine Learning Research](https://mlsanity.com/paper/2302.04054/) (2023)
- [Questionable practices in machine learning](https://mlsanity.com/paper/2407.12220/) (2024)
- [Not All Explanations for Deep Learning Phenomena Are Equally Valuable](https://mlsanity.com/paper/2506.23286/) (2025)
- [Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events](https://mlsanity.com/paper/2506.06808/) (2025)
- [MetaRSI / RSI2: A Meta-Recursive Self-Improving System for Recursive Self-Improving Systems Themselves](https://mlsanity.com/paper/2609.06396/) (2026)
- [Bringing Atomistic Deep Learning to Prime Time](https://mlsanity.com/paper/2112.04977/) (2021)
- [Causal Structure is Inducible but Functionally Decoupled: The Routing/Readout Boundary of a Typed Mechanism Library](https://mlsanity.com/paper/2608.11767/) (2026)

## Abstract (arXiv)

In the quest to align deep learning with the sciences to address calls for rigor, safety, and interpretability in machine learning systems, this contribution identifies key missing pieces: the stages of hypothesis formulation and testing, as well as statistical and systematic uncertainty estimation -- core tenets of the scientific method. This position paper discusses the ways in which contemporary science is conducted in other domains and identifies potentially useful practices. We present a case study from physics and describe how this field has promoted rigor through specific methodological practices, and provide recommendations on how machine learning researchers can adopt these practices into the research ecosystem. We argue that both domain-driven experiments and application-agnostic questions of the inner workings of fundamental building blocks of machine learning models ought to be examined with the tools of the scientific method, to ensure we not only understand effect, but also begin to understand cause, which is the raison d'etre of science.