# The NCI Imaging Data Commons as a platform for reproducible research in computational pathology

arXiv: https://arxiv.org/abs/2303.09354 · HTML: https://mlsanity.com/paper/2303.09354/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 55 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 6 of 6)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Deterministic Implementations for Reproducibility in Deep Reinforcement Learning](https://mlsanity.com/paper/1809.05676/) (2018)
- [Recommendations on test datasets for evaluating AI solutions in pathology](https://mlsanity.com/paper/2204.14226/) (2022)

## Built on by (most cited first, 4 of 4)

- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [Whole Slide Concepts: A Supervised Foundation Model For Pathological Images](https://mlsanity.com/paper/2507.05742/) (2025)
- [Boostlet.js: Image processing plugins for the web via JavaScript injection](https://mlsanity.com/paper/2405.07868/) (2024)
- [Sharing standardized image-derived data in computational pathology using DICOM](https://mlsanity.com/paper/2609.14530/) (2026)

## Abstract (arXiv)

Background and Objectives: Reproducibility is a major challenge in developing machine learning (ML)-based solutions in computational pathology (CompPath). The NCI Imaging Data Commons (IDC) provides >120 cancer image collections according to the FAIR principles and is designed to be used with cloud ML services. Here, we explore its potential to facilitate reproducibility in CompPath research. Methods: Using the IDC, we implemented two experiments in which a representative ML-based method for classifying lung tumor tissue was trained and/or evaluated on different datasets. To assess reproducibility, the experiments were run multiple times with separate but identically configured instances of common ML services. Results: The AUC values of different runs of the same experiment were generally consistent. However, we observed small variations in AUC values of up to 0.045, indicating a practical limit to reproducibility. Conclusions: We conclude that the IDC facilitates approaching the reproducibility limit of CompPath research (i) by enabling researchers to reuse exactly the same datasets and (ii) by integrating with cloud ML services so that experiments can be run in identically configured computing environments.