mlsanityData to 14 Oct 2026

REFORMS: Reporting Standards for Machine Learning Based Science

arXiv 2308.07832 ↗First posted 15 Aug 2023cs.LG cs.AI stat.MEcited by 13

Not on the map. Why?

Abstract (arXiv)

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear reporting standards for ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (Reporting Standards For Machine Learning Based Science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed based on a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 38 of its 190 references that are in the map. First posted on arXiv 15 Aug 2023.

Builds on

38 of 190 references.

References · 38

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  3. 2018Datasheets for Datasetscited by 1,386
  4. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  5. 2018Model Cards for Model Reportingcited by 1,080
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  8. 2020A Metric Learning Reality Checkcited by 321
  9. 2019Measurement and Fairnesscited by 308
  10. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  11. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  12. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
26 more
  1. 2019Explainable Machine Learning for Scientific Insights and Discoveriescited by 166
  2. 2019Predictive Multiplicity in Classificationcited by 166
  3. 2020Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scanscited by 158
  4. 2018Model Evaluation, Model Selection, and Algorithm Selection in Machine Learningcited by 157
  5. 2020Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertaintycited by 129
  6. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 119
  7. 2018Tunability: Importance of Hyperparameters of Machine Learning Algorithmscited by 99
  8. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  9. 2022The Fallacy of AI Functionalitycited by 88
  10. 2019A Step Toward Quantifying Independently Reproducible Machine Learning Researchcited by 77
  11. 2022Leakage and the Reproducibility Crisis in ML-based Sciencecited by 65
  12. 2019Machine Learning Methods Economists Should Know Aboutcited by 64
  13. 2023Benchmarking Neural Network Training Algorithmscited by 52
  14. 2021Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Paperscited by 49
  15. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  16. 2022Predictive Multiplicity in Probabilistic Classificationcited by 44
  17. 2021How to avoid machine learning pitfalls: a guide for academic researcherscited by 35
  18. 2022Classification of datasets with imputed missing values: does imputation quality matter?cited by 22
  19. 2020A Hierarchy of Limitations in Machine Learningcited by 20
  20. 2021Hyperparameter Optimization Is Deceiving Us, and How to Stop Itcited by 20
  21. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  22. 2021Perspectives on Machine Learning from Psychology's Reproducibility Crisiscited by 9
  23. 2021Model Selection's Disparate Impact in Real-World Deep Learning Applicationscited by 7
  24. 2013The availability of research data declines rapidly with article agecited by 2
  25. 2017The Theory is Predictive, but is it Complete? An Application to Human Perception of Randomnesscited by 2
  26. 2022Navigating the challenges in creating complex data systems: a development philosophycited by 2

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 6; 2025: 3.

13 papers.

Sort:

  1. 2024Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equationscited by 91
  2. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  3. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 46
  4. 2023Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligencecited by 15
  5. 2023Pre-registration for Predictive Modelingcited by 13
  6. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  7. 2024Unraveling overoptimism and publication bias in ML-driven sciencecited by 3
  8. 2024Multi-Sensor Deep Learning for Glacier Mappingcited by 2
  9. 2023Constructing Impactful Machine Learning Research for Astronomy: Best Practices for Researchers and Reviewerscited by 1
  10. 2024From Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility Gapcited by 1
  11. 2025Review and Recommendations for using Artificial Intelligence in Intracoronary Optical Coherence Tomography Analysiscited by 0
  12. 2025Temporal Image Forensics: A Review and Critical Evaluationcited by 0
1 more
  1. 2025Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Networkcited by 0