mlsanityData to 7 Oct 2026

Pitfalls in Machine Learning Research: Reexamining the Development Cycle

arXiv 2011.02832 ↗First posted 4 Nov 2020cs.LG stat.ME stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

Machine learning has the potential to fuel further advances in data science, but it is greatly hindered by an ad hoc design process, poor data hygiene, and a lack of statistical rigor in model evaluation. Recently, these issues have begun to attract more attention as they have caused public and embarrassing issues in research and development. Drawing from our experience as machine learning researchers, we follow the machine learning process from algorithm design to data collection to model evaluation, drawing attention to common pitfalls and providing practical recommendations for improvements. At each step, case studies are introduced to highlight how these pitfalls occur in practice, and where things could be improved.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 4; 2023: 5; 2024: 3; 2025: 1; 2026: 1.

17 papers.

Sort:

By area:

  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  2. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  3. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  4. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  5. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  6. 2025No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasetscited by 16
  7. 2021Cut the CARP: Fishing for zero-shot story evaluationcited by 14
  8. 2022Fooling MOSS Detection with Pretrained Language Modelscited by 12
  9. 2024Benchmark Data Repositories for Better Benchmarkingcited by 10
  10. 2023What are the Machine Learning best practices reported by practitioners on Stack Exchange?cited by 3
  11. 2023NeuroCodeBench: a plain C neural network benchmark for software verificationcited by 3
  12. 2024Automated Repair of AI Code with Large Language Models and Formal Verificationcited by 3
5 more
  1. 2021Life is not black and white -- Combining Semi-Supervised Learning with fuzzy labelscited by 0
  2. 2022Leveraging Centric Data Federated Learning Using Blockchain For Integrity Assurancecited by 0
  3. 2023Towards machine learning guided by best practicescited by 0
  4. 2024Perspective of Software Engineering Researchers on Machine Learning Practices Regarding Research, Review, and Educationcited by 0
  5. 2026To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systemscited by 0