# Troubling Trends in Machine Learning Scholarship

arXiv: https://arxiv.org/abs/1807.03341 · HTML: https://mlsanity.com/paper/1807.03341/

## Key facts

- Cited by 108 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2018.

## Built on by (most cited first, 30 of 108)

- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Dive into Deep Learning](https://mlsanity.com/paper/2106.11342/) (2021)
- [Art and the science of generative AI: A deeper dive](https://mlsanity.com/paper/2306.04141/) (2023)
- [A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility](https://mlsanity.com/paper/2504.07086/) (2025)
- [GraphWorld: Fake Graphs Bring Real Insights for GNNs](https://mlsanity.com/paper/2203.00112/) (2022)
- [Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging](https://mlsanity.com/paper/2209.14981/) (2022)
- [A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification](https://mlsanity.com/paper/2211.15259/) (2022)
- [A Review of Biomedical Datasets Relating to Drug Discovery: A Knowledge Graph Perspective](https://mlsanity.com/paper/2102.10062/) (2021)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [Revisiting RCAN: Improved Training for Image Super-Resolution](https://mlsanity.com/paper/2201.11279/) (2022)
- [Sources of Irreproducibility in Machine Learning: A Review](https://mlsanity.com/paper/2204.07610/) (2022)
- [Considerations for meaningful sign language machine translation based on glosses](https://mlsanity.com/paper/2211.15464/) (2022)
- [Application-Driven Innovation in Machine Learning](https://mlsanity.com/paper/2403.17381/) (2024)
- [Marrying Fairness and Explainability in Supervised Learning](https://mlsanity.com/paper/2204.02947/) (2022)
- [Hyperparameter Optimization Is Deceiving Us, and How to Stop It](https://mlsanity.com/paper/2102.03034/) (2021)
- [Position: Why We Must Rethink Empirical Research in Machine Learning](https://mlsanity.com/paper/2405.02200/) (2024)
- [The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning](https://mlsanity.com/paper/2203.06498/) (2022)
- [Deep Learning and Traffic Classification: Lessons learned from a commercial-grade dataset with hundreds of encrypted and zero-day applications](https://mlsanity.com/paper/2104.03182/) (2021)
- [Pre-registration for Predictive Modeling](https://mlsanity.com/paper/2311.18807/) (2023)
- [No One Knows the State of the Art in Geospatial Foundation Models](https://mlsanity.com/paper/2605.12678/) (2026)
- [Modeling the Machine Learning Multiverse](https://mlsanity.com/paper/2206.05985/) (2022)
- [The 2025 Foundation Model Transparency Index](https://mlsanity.com/paper/2512.10169/) (2025)
- [GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models](https://mlsanity.com/paper/2602.06718/) (2026)
- [GraphGen-Redux: a Fast and Lightweight Recurrent Model for labeled Graph Generation](https://mlsanity.com/paper/2107.08396/) (2021)
- [Benchmark Data Repositories for Better Benchmarking](https://mlsanity.com/paper/2410.24100/) (2024)

## Abstract (arXiv)

Collectively, machine learning (ML) researchers are engaged in the creation and dissemination of knowledge about data-driven algorithms. In a given paper, researchers might aspire to any subset of the following goals, among others: to theoretically characterize what is learnable, to obtain understanding through empirically rigorous experiments, or to build a working system that has high predictive accuracy. While determining which knowledge warrants inquiry may be subjective, once the topic is fixed, papers are most valuable to the community when they act in service of the reader, creating foundational knowledge and communicating as clearly as possible. Recent progress in machine learning comes despite frequent departures from these ideals. In this paper, we focus on the following four patterns that appear to us to be trending in ML scholarship: (i) failure to distinguish between explanation and speculation; (ii) failure to identify the sources of empirical gains, e.g., emphasizing unnecessary modifications to neural architectures when gains actually stem from hyper-parameter tuning; (iii) mathiness: the use of mathematics that obfuscates or impresses rather than clarifies, e.g., by confusing technical and non-technical concepts; and (iv) misuse of language, e.g., by choosing terms of art with colloquial connotations or by overloading established technical terms. While the causes behind these patterns are uncertain, possibilities include the rapid expansion of the community, the consequent thinness of the reviewer pool, and the often-misaligned incentives between scholarship and short-term measures of success (e.g., bibliometrics, attention, and entrepreneurial opportunity). While each pattern offers a corresponding remedy (don't do it), we also discuss some speculative suggestions for how the community might combat these trends.