mlsanityData to 7 Oct 2026

ImageNot: A contrast with ImageNet preserves model rankings

arXiv 2404.02112 ↗First posted 2 Apr 2024cs.LG cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

We introduce ImageNot, a dataset constructed explicitly to be drastically different than ImageNet while matching its scale. ImageNot is designed to test the external validity of deep learning progress on ImageNet. We show that key model architectures developed for ImageNet over the years rank identically to how they rank on ImageNet when trained from scratch and evaluated on ImageNot. Moreover, the relative improvements of each model over earlier models strongly correlate in both datasets. Our work demonstrates a surprising degree of external validity in the relative performance of image classification models when trained and evaluated on an entirely different dataset. This stands in contrast with absolute accuracy numbers that typically drop sharply even under small changes to a dataset.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 32 of its 53 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

32 of 53 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2017Squeeze-and-Excitation Networkscited by 4,763
  12. 2014Going Deeper with Convolutionscited by 4,231
20 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2022A ConvNet for the 2020scited by 3,545
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2019Searching for MobileNetV3cited by 1,984
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2018Do Better ImageNet Models Transfer Better?cited by 560
  9. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  10. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  11. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  12. 2020Are we done with ImageNet?cited by 329
  13. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  14. 2018A System for Massively Parallel Hyperparameter Tuningcited by 201
  15. 2016Speed/accuracy trade-offs for modern convolutional object detectorscited by 198
  16. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  17. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85
  18. 2024A Decade's Battle on Dataset Bias: Are We There Yet?cited by 46
  19. 2023What Makes ImageNet Look Unlike LAIONcited by 9
  20. 2024Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budgetcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 5.

14 papers.

Sort:

By area:

  1. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  2. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  3. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 13
  4. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  5. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 12
  6. 2024ACES: Automatic Cohort Extraction System for Event-Stream Datasetscited by 8
  7. 2024What Constitutes a Less Discriminatory Algorithm?cited by 8
  8. 2025Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests insteadcited by 7
  9. 2025We Need to Rethink Benchmarking in Anomaly Detectioncited by 4
  10. 2026SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Casescited by 3
  11. 2026Leaderboard Incentives: Model Rankings under Strategic Post-Trainingcited by 2
  12. 2026Aligning Language Model Benchmarks with Pairwise Preferencescited by 0
2 more
  1. 2026Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfercited by 0
  2. 2026Rethinking How We Evaluate Methodological Progress in Health AIcited by 0