mlsanityData to 7 Oct 2026

How to avoid machine learning pitfalls: a guide for academic researchers

arXiv 2108.02497 ↗First posted 5 Aug 2021cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Mistakes in machine learning practice are commonplace, and can result in a loss of confidence in the findings and products of machine learning. This guide outlines common mistakes that occur when using machine learning, and what can be done to avoid them. Whilst it should be accessible to anyone with a basic understanding of machine learning techniques, it focuses on issues that are of particular concern within academic research, such as the need to do rigorous comparisons and reach valid conclusions. It covers five stages of the machine learning process: what to do before model building, how to reliably build models, how to robustly evaluate models, how to compare models fairly, and how to report results.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 26 of its 83 references that are in the map. First posted on arXiv 5 Aug 2021.

Builds on

26 of 83 references.

References · 26

  1. 2021Transformers in Vision: A Surveycited by 1,009
  2. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  3. 2021A Survey of Transformerscited by 399
  4. 2020A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospectscited by 396
  5. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  6. 2020Fairness in Machine Learning: A Surveycited by 369
  7. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  8. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 318
  9. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  10. 2020On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practicecited by 181
  11. 2020Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scanscited by 158
  12. 2018Model Evaluation, Model Selection, and Algorithm Selection in Machine Learningcited by 157
14 more
  1. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  2. 2020A Survey on Evolutionary Neural Architecture Searchcited by 142
  3. 2020An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networkscited by 128
  4. 2021Dive into Deep Learningcited by 111
  5. 2020Dos and Don'ts of Machine Learning in Computer Securitycited by 98
  6. 2016Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysiscited by 92
  7. 2016Understanding data augmentation for classification: when to warp?cited by 74
  8. 2020General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Modelscited by 74
  9. 2019Ablation Studies in Artificial Neural Networkscited by 58
  10. 2019Evaluating time series forecasting models: An empirical study on performance estimation methodscited by 52
  11. 2020A Hierarchy of Limitations in Machine Learningcited by 19
  12. 2020Recent Trends in the Use of Statistical Tests for Comparing Swarm and Evolutionary Computing Algorithms: Practical Guidelines and a Critical Reviewcited by 14
  13. 2020A critical analysis of metrics used for measuring progress in artificial intelligencecited by 14
  14. 2020MLJ: A Julia package for composable machine learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 7; 2024: 11; 2025: 5; 2026: 9.

34 papers.

Sort:

  1. 2022Leakage and the Reproducibility Crisis in ML-based Sciencecited by 63
  2. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  3. 2024EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurationscited by 14
  4. 2023REFORMS: Reporting Standards for Machine Learning Based Sciencecited by 13
  5. 2024A Guide to Feature Importance Methods for Scientific Inferencecited by 11
  6. 2024Estimating Model Performance Under Covariate Shift Without Labelscited by 9
  7. 2025A validity-guided workflow for robust large language model research in psychologycited by 8
  8. 2023Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Developmentcited by 6
  9. 2025Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoptioncited by 6
  10. 2024Anytime-Valid Inference for Double/Debiased Machine Learning of Causal Parameterscited by 5
  11. 2023Stable Linear Subspace Identification: A Machine Learning Approachcited by 4
  12. 2023What are the Machine Learning best practices reported by practitioners on Stack Exchange?cited by 3
22 more
  1. 2024Questionable practices in machine learningcited by 3
  2. 2024Map-Based Path Loss Prediction in Multiple Cities Using Convolutional Neural Networkscited by 3
  3. 2025Towards a more realistic evaluation of machine learning models for bearing fault diagnosiscited by 2
  4. 2023Reassessing feature-based Android malware detection in a contemporary contextcited by 1
  5. 2026Clever Materials: When Models Identify Good Materials for the Wrong Reasonscited by 1
  6. 2026Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Paperscited by 1
  7. 2021Small data problems in political research: a critical replication studycited by 0
  8. 2023Towards a User Privacy-Aware Mobile Gaming App Installation Prediction Modelcited by 0
  9. 2023Online ML Self-adaptation in Face of Trapscited by 0
  10. 2024BotSSCL: Social Bot Detection with Self-Supervised Contrastive Learningcited by 0
  11. 2024Designing deep neural networks for driver intention recognitioncited by 0
  12. 2024Machine learning meets mass spectrometry: a focused perspectivecited by 0
  13. 2024Perspective of Software Engineering Researchers on Machine Learning Practices Regarding Research, Review, and Educationcited by 0
  14. 2025Environmental Feature Engineering and Statistical Validation for ML-Based Path Loss Predictioncited by 0
  15. 2025Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Educationcited by 0
  16. 2026Which Leakage Types Matter? A Quantitative Landscape Across 2,047 Benchmark Datasetscited by 0
  17. 2026To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systemscited by 0
  18. 2026Survey-aware Machine Learning: A Guideline for Valid Population Health Inference based on Scoping Reviewcited by 0
  19. 2026Beyond the Gegenbauer Paradigm: q-Orthogonal Kernels for Machine Learningcited by 0
  20. 2026Beyond Measurement Metrics: A Human-Centered Framework for Semantic Validation of Network Traffic Classificationcited by 0
  21. 2026Artificial Neural Network Assisted Modelling of Tangent Galvanometer Measurements for the Determination of Horizontal Component of Earth's Magnetic Fieldcited by 0
  22. 2026Multi-Class, Multi-Tier Network Intrusion Detection: A Comprehensive and Reproducible Benchmarkcited by 0