# Which Leakage Types Matter? A Quantitative Landscape Across 2,047 Benchmark Datasets

arXiv: https://arxiv.org/abs/2604.04199 · HTML: https://mlsanity.com/paper/2604.04199/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 49 references that are in the map.
- First posted on arXiv 5 Apr 2026.

## Builds on (most cited first, 11 of 11)

- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Cross-validation failure: small sample sizes lead to large error bars](https://mlsanity.com/paper/1706.07581/) (2017)
- [How to avoid machine learning pitfalls: a guide for academic researchers](https://mlsanity.com/paper/2108.02497/) (2021)
- [Bootstrapping the Out-of-sample Predictions for Efficient and Accurate Cross-Validation](https://mlsanity.com/paper/1708.07180/) (2017)
- [Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning](https://mlsanity.com/paper/2401.13796/) (2024)
- [On Leakage in Machine Learning Pipelines](https://mlsanity.com/paper/2311.04179/) (2023)
- [LeakageDetector 2.0: Analyzing Data Leakage in Jupyter-Driven Machine Learning Pipelines](https://mlsanity.com/paper/2509.15971/) (2025)
- [A Grammar of Machine Learning Workflows: Rejecting Data Leakage at Call Time](https://mlsanity.com/paper/2603.10742/) (2026)

## Abstract (arXiv)

Twenty-eight within-subject counterfactual experiments across 2,047 iid tabular datasets, plus a boundary experiment on 129 temporal datasets, measure the severity of four data leakage classes in machine learning. Class I (estimation: fitting scalers on full data) is negligible: all nine conditions produce |DeltaAUC| ≤ 0.005. Class II (selection: peeking, seed cherry-picking) is substantial: the measured effect is consistent with about 90% noise exploitation inflating reported scores. Class III (memorization) scales with model capacity: dz = 0.37 (Naive Bayes) to 1.11 (Decision Tree) at 10% duplication. Class IV (boundary) is invisible under random cross-validation. Within this iid tabular regime, the textbook emphasis is inverted: normalization leakage matters least; selection leakage at practical dataset sizes matters most.