mlsanityData to 14 Oct 2026

Model Similarity Mitigates Test Set Overuse

arXiv 1905.12580 ↗First posted 29 May 2019cs.LG stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

Excessive reuse of test data has become commonplace in today's machine learning workflows. Popular benchmarks, competitions, industrial scale tuning, among other applications, all involve test data reuse beyond guidance by statistical confidence bounds. Nonetheless, recent replication studies give evidence that popular benchmarks continue to support progress despite years of extensive reuse. We proffer a new explanation for the apparent longevity of test data: Many proposed models are similar in their predictions and we prove that this similarity mitigates overfitting. Specifically, we show empirically that models proposed for the ImageNet ILSVRC benchmark agree in their predictions well beyond what we can conclude from their accuracy levels alone. Likewise, models created by large scale hyperparameter search enjoy high levels of similarity. Motivated by these empirical observations, we give a non-asymptotic generalization bound that takes similarity into account, leading to meaningful confidence bounds in practical settings.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 11; 2023: 8; 2024: 4; 2025: 2; 2026: 5.

38 papers.

Sort:

By area:

  1. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  2. 2021Partial success in closing the gap between human and machine visioncited by 185
  3. 2023Getting aligned on representational alignmentcited by 143
  4. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  5. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 75
  6. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 61
  7. 2021No One Representation to Rule Them All: Overlapping Features of Training Methodscited by 58
  8. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 51
  9. 2022Mechanistic Mode Connectivitycited by 50
  10. 2022Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioningcited by 45
  11. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 37
  12. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
26 more
  1. 2022ModelDiff: A Framework for Comparing Learning Algorithmscited by 30
  2. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 28
  3. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 28
  4. 2022Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classificationcited by 26
  5. 2021Forward Compatible Training for Large-Scale Embedding Retrieval Systemscited by 19
  6. 2022Deconstructing Distributions: A Pointwise Framework of Learningcited by 19
  7. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
  8. 2024How Much Can We Forget about Data Contamination?cited by 14
  9. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  10. 2023A Holistic Assessment of the Reliability of Machine Learning Systemscited by 9
  11. 2021Rip van Winkle's Razor: A Simple Estimate of Overfit to Test Datacited by 8
  12. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 7
  13. 2023A Simple and Efficient Baseline for Data Attribution on Imagescited by 5
  14. 2022Similarity of Neural Architectures using Adversarial Attack Transferabilitycited by 4
  15. 2026Leaderboard Incentives: Model Rankings under Strategic Post-Trainingcited by 3
  16. 2026What Fits (Into Few Tokens) Doesn't Overfit: Compression and Generalization in ML Research Agentscited by 3
  17. 2023Revisiting Hidden Representations in Transfer Learning for Medical Imagingcited by 2
  18. 2023Inventing a coin-flip classifier: Accounting for multiplicity in machine learning benchmark performancecited by 2
  19. 2026Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similaritycited by 2
  20. 2022FBI: Fingerprinting models with Benign Inputscited by 1
  21. 2023Multi-Symmetry Ensembles: Improving Diversity and Generalization via Opposing Symmetriescited by 1
  22. 2021When Neural Networks Using Different Sensors Create Similar Featurescited by 0
  23. 2022Sequential algorithmic modification with test data reusecited by 0
  24. 2023On Statistical Learning of Branch and Bound for Vehicle Routing Optimizationcited by 0
  25. 2026How Reusable Are Benchmarks with Richer Feedback?cited by 0
  26. 2026On Hyperparameter Tuning on the Test Setcited by 0