mlsanityData to 14 Oct 2026

Reproducibility Requires Consolidated Artifacts

arXiv 2305.12571 ↗First posted 21 May 2023cs.LG cs.AI cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

Machine learning is facing a 'reproducibility crisis' where a significant number of works report failures when attempting to reproduce previously published results. We evaluate the sources of reproducibility failures using a meta-analysis of 142 replication studies from ReScience C and 204 code repositories. We find that missing experiment details such as hyperparameters are potential causes of unreproducibility. We experimentally show the bias of different hyperparameter selection strategies and conclude that consolidated artifacts with a unified framework can help support reproducibility.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 6 of its 9 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

6 of 9 references.

References · 6

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  3. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  4. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
  5. 2020NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Sizecited by 98
  6. 2021Emulation of physical processes with Emukitcited by 24

Built on by

Not cited yet.