# Pitfalls in Machine Learning Research: Reexamining the Development Cycle

arXiv: https://arxiv.org/abs/2011.02832 · HTML: https://mlsanity.com/paper/2011.02832/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2020.

## Built on by (most cited first, 17 of 17)

- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://mlsanity.com/paper/2306.16527/) (2023)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [The MiniPile Challenge for Data-Efficient Language Models](https://mlsanity.com/paper/2304.08442/) (2023)
- [BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing](https://mlsanity.com/paper/2206.15076/) (2022)
- [No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets](https://mlsanity.com/paper/2502.02379/) (2025)
- [Cut the CARP: Fishing for zero-shot story evaluation](https://mlsanity.com/paper/2110.03111/) (2021)
- [Fooling MOSS Detection with Pretrained Language Models](https://mlsanity.com/paper/2201.07406/) (2022)
- [Benchmark Data Repositories for Better Benchmarking](https://mlsanity.com/paper/2410.24100/) (2024)
- [What are the Machine Learning best practices reported by practitioners on Stack Exchange?](https://mlsanity.com/paper/2301.10516/) (2023)
- [NeuroCodeBench: a plain C neural network benchmark for software verification](https://mlsanity.com/paper/2309.03617/) (2023)
- [Automated Repair of AI Code with Large Language Models and Formal Verification](https://mlsanity.com/paper/2405.08848/) (2024)
- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)
- [Life is not black and white -- Combining Semi-Supervised Learning with fuzzy labels](https://mlsanity.com/paper/2110.06592/) (2021)
- [Leveraging Centric Data Federated Learning Using Blockchain For Integrity Assurance](https://mlsanity.com/paper/2206.04731/) (2022)
- [Towards machine learning guided by best practices](https://mlsanity.com/paper/2305.00233/) (2023)
- [Perspective of Software Engineering Researchers on Machine Learning Practices Regarding Research, Review, and Education](https://mlsanity.com/paper/2411.19304/) (2024)

## Abstract (arXiv)

Machine learning has the potential to fuel further advances in data science, but it is greatly hindered by an ad hoc design process, poor data hygiene, and a lack of statistical rigor in model evaluation. Recently, these issues have begun to attract more attention as they have caused public and embarrassing issues in research and development. Drawing from our experience as machine learning researchers, we follow the machine learning process from algorithm design to data collection to model evaluation, drawing attention to common pitfalls and providing practical recommendations for improvements. At each step, case studies are introduced to highlight how these pitfalls occur in practice, and where things could be improved.