# Identifying Statistical Bias in Dataset Replication

arXiv: https://arxiv.org/abs/2005.09619 · HTML: https://mlsanity.com/paper/2005.09619/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2020.

## Built on by (most cited first, 24 of 24)

- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Blind Baselines Beat Membership Inference Attacks for Foundation Models](https://mlsanity.com/paper/2406.16201/) (2024)
- [RobustNav: Towards Benchmarking Robustness in Embodied Navigation](https://mlsanity.com/paper/2106.04531/) (2021)
- [Describing Differences in Image Sets with Natural Language](https://mlsanity.com/paper/2312.02974/) (2023)
- [A Decade's Battle on Dataset Bias: Are We There Yet?](https://mlsanity.com/paper/2403.08632/) (2024)
- [Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generation](https://mlsanity.com/paper/2302.07865/) (2023)
- [3DB: A Framework for Debugging Computer Vision Models](https://mlsanity.com/paper/2106.03805/) (2021)
- [Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability](https://mlsanity.com/paper/2106.07393/) (2021)
- [On Modality Bias Recognition and Reduction](https://mlsanity.com/paper/2202.12690/) (2022)
- [The statistical advantage of automatic NLG metrics at the system level](https://mlsanity.com/paper/2105.12437/) (2021)
- [A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labels](https://mlsanity.com/paper/2109.11126/) (2021)
- [What Makes ImageNet Look Unlike LAION](https://mlsanity.com/paper/2306.15769/) (2023)
- [A Siren Song of Open Source Reproducibility](https://mlsanity.com/paper/2204.04372/) (2022)
- [Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts](https://mlsanity.com/paper/2107.12052/) (2021)
- [What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?](https://mlsanity.com/paper/2406.09908/) (2024)
- [Generative multitask learning mitigates target-causing confounding](https://mlsanity.com/paper/2202.04136/) (2022)
- [Quantitatively Measuring and Contrastively Exploring Heterogeneity for Domain Generalization](https://mlsanity.com/paper/2305.15889/) (2023)
- [Leveraging Human-Machine Interactions for Computer Vision Dataset Quality Enhancement](https://mlsanity.com/paper/2401.17736/) (2024)
- [Defuse: Harnessing Unrestricted Adversarial Examples for Debugging Models Beyond Test Accuracy](https://mlsanity.com/paper/2102.06162/) (2021)
- [A Principled Evaluation Protocol for Comparative Investigation of the Effectiveness of DNN Classification Models on Similar-but-non-identical Datasets](https://mlsanity.com/paper/2209.01848/) (2022)
- [Self-Improving-Leaderboard(SIL): A Call for Real-World Centric Natural Language Processing Leaderboards](https://mlsanity.com/paper/2303.10888/) (2023)
- [Leveraging generative models to characterize the failure conditions of image classifiers](https://mlsanity.com/paper/2410.12814/) (2024)
- [The Impact of the Single-Label Assumption in Image Recognition Benchmarking](https://mlsanity.com/paper/2412.18409/) (2024)
- [Diverse, Difficult, and Odd Instances (D2O): A New Test Set for Object Classification](https://mlsanity.com/paper/2301.12527/) (2023)

## Abstract (arXiv)

Dataset replication is a useful tool for assessing whether improvements in test accuracy on a specific benchmark correspond to improvements in models' ability to generalize reliably. In this work, we present unintuitive yet significant ways in which standard approaches to dataset replication introduce statistical bias, skewing the resulting observations. We study ImageNet-v2, a replication of the ImageNet dataset on which models exhibit a significant (11-14%) drop in accuracy, even after controlling for a standard human-in-the-loop measure of data quality. We show that after correcting for the identified statistical bias, only an estimated 3.6% ± 1.5% of the original 11.7% ± 1.0% accuracy drop remains unaccounted for. We conclude with concrete recommendations for recognizing and avoiding bias in dataset replication. Code for our study is publicly available at http://github.com/MadryLab/dataset-replication-analysis .