# How to avoid machine learning pitfalls: a guide for academic researchers

arXiv: https://arxiv.org/abs/2108.02497 · HTML: https://mlsanity.com/paper/2108.02497/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 83 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 26 of 26)

- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects](https://mlsanity.com/paper/2004.02806/) (2020)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice](https://mlsanity.com/paper/2007.15745/) (2020)
- [Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans](https://mlsanity.com/paper/2008.06388/) (2020)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [A Survey on Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2008.10937/) (2020)
- [An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks](https://mlsanity.com/paper/2007.15951/) (2020)
- [Dive into Deep Learning](https://mlsanity.com/paper/2106.11342/) (2021)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models](https://mlsanity.com/paper/2007.04131/) (2020)
- [Understanding data augmentation for classification: when to warp?](https://mlsanity.com/paper/1609.08764/) (2016)
- [Ablation Studies in Artificial Neural Networks](https://mlsanity.com/paper/1901.08644/) (2019)
- [Evaluating time series forecasting models: An empirical study on performance estimation methods](https://mlsanity.com/paper/1905.11744/) (2019)
- [A Hierarchy of Limitations in Machine Learning](https://mlsanity.com/paper/2002.05193/) (2020)
- [Recent Trends in the Use of Statistical Tests for Comparing Swarm and Evolutionary Computing Algorithms: Practical Guidelines and a Critical Review](https://mlsanity.com/paper/2002.09227/) (2020)
- [A critical analysis of metrics used for measuring progress in artificial intelligence](https://mlsanity.com/paper/2008.02577/) (2020)
- [MLJ: A Julia package for composable machine learning](https://mlsanity.com/paper/2007.12285/) (2020)

## Built on by (most cited first, 30 of 35)

- [Leakage and the Reproducibility Crisis in ML-based Science](https://mlsanity.com/paper/2207.07048/) (2022)
- [Position: Why We Must Rethink Empirical Research in Machine Learning](https://mlsanity.com/paper/2405.02200/) (2024)
- [EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurations](https://mlsanity.com/paper/2402.00491/) (2024)
- [REFORMS: Reporting Standards for Machine Learning Based Science](https://mlsanity.com/paper/2308.07832/) (2023)
- [A Guide to Feature Importance Methods for Scientific Inference](https://mlsanity.com/paper/2404.12862/) (2024)
- [Estimating Model Performance Under Covariate Shift Without Labels](https://mlsanity.com/paper/2401.08348/) (2024)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Development](https://mlsanity.com/paper/2307.11525/) (2023)
- [Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption](https://mlsanity.com/paper/2508.08883/) (2025)
- [Anytime-Valid Inference for Double/Debiased Machine Learning of Causal Parameters](https://mlsanity.com/paper/2408.09598/) (2024)
- [Stable Linear Subspace Identification: A Machine Learning Approach](https://mlsanity.com/paper/2311.03197/) (2023)
- [Questionable practices in machine learning](https://mlsanity.com/paper/2407.12220/) (2024)
- [Towards a more realistic evaluation of machine learning models for bearing fault diagnosis](https://mlsanity.com/paper/2509.22267/) (2025)
- [What are the Machine Learning best practices reported by practitioners on Stack Exchange?](https://mlsanity.com/paper/2301.10516/) (2023)
- [Map-Based Path Loss Prediction in Multiple Cities Using Convolutional Neural Networks](https://mlsanity.com/paper/2411.17752/) (2024)
- [Clever Materials: When Models Identify Good Materials for the Wrong Reasons](https://mlsanity.com/paper/2602.17730/) (2026)
- [Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education](https://mlsanity.com/paper/2505.18220/) (2025)
- [Reassessing feature-based Android malware detection in a contemporary context](https://mlsanity.com/paper/2301.12778/) (2023)
- [Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Papers](https://mlsanity.com/paper/2603.06862/) (2026)
- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)
- [Small data problems in political research: a critical replication study](https://mlsanity.com/paper/2109.12911/) (2021)
- [Towards a User Privacy-Aware Mobile Gaming App Installation Prediction Model](https://mlsanity.com/paper/2302.03332/) (2023)
- [Online ML Self-adaptation in Face of Traps](https://mlsanity.com/paper/2309.05805/) (2023)
- [BotSSCL: Social Bot Detection with Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2402.03740/) (2024)
- [Designing deep neural networks for driver intention recognition](https://mlsanity.com/paper/2402.05150/) (2024)
- [Machine learning meets mass spectrometry: a focused perspective](https://mlsanity.com/paper/2407.00117/) (2024)
- [Perspective of Software Engineering Researchers on Machine Learning Practices Regarding Research, Review, and Education](https://mlsanity.com/paper/2411.19304/) (2024)
- [Environmental Feature Engineering and Statistical Validation for ML-Based Path Loss Prediction](https://mlsanity.com/paper/2501.08306/) (2025)
- [Which Leakage Types Matter? A Quantitative Landscape Across 2,047 Benchmark Datasets](https://mlsanity.com/paper/2604.04199/) (2026)
- [Survey-aware Machine Learning: A Guideline for Valid Population Health Inference based on Scoping Review](https://mlsanity.com/paper/2605.08963/) (2026)

## Abstract (arXiv)

Mistakes in machine learning practice are commonplace, and can result in a loss of confidence in the findings and products of machine learning. This guide outlines common mistakes that occur when using machine learning, and what can be done to avoid them. Whilst it should be accessible to anyone with a basic understanding of machine learning techniques, it focuses on issues that are of particular concern within academic research, such as the need to do rigorous comparisons and reach valid conclusions. It covers five stages of the machine learning process: what to do before model building, how to reliably build models, how to robustly evaluate models, how to compare models fairly, and how to report results.