mlsanityData to 7 Oct 2026

Does progress on ImageNet transfer to real-world datasets?

arXiv 2301.04644 ↗First posted 11 Jan 2023cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Does progress on ImageNet transfer to real-world datasets? We investigate this question by evaluating ImageNet pre-trained models with varying accuracy (57% - 83%) on six practical image classification datasets. In particular, we study datasets collected with the goal of solving real-world tasks (e.g., classifying images from camera traps or satellites), as opposed to web-scraped benchmarks collected for comparing models. On multiple datasets, models with higher ImageNet accuracy do not consistently yield performance improvements. For certain tasks, interventions such as data augmentation improve performance even when architectures do not. We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 59 of its 87 references that are in the map. First posted on arXiv 11 Jan 2023.

Builds on

59 of 87 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
38 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  2. 2019Searching for MobileNetV3cited by 1,984
  3. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2017Random Erasing Data Augmentationcited by 1,235
  9. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  10. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  11. 2021Scaling Vision Transformerscited by 993
  12. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  13. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  14. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  15. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  16. 2018Recognition in Terra Incognitacited by 655
  17. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  18. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  19. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  20. 2018Do Better ImageNet Models Transfer Better?cited by 560
  21. 2017Progressive Neural Architecture Searchcited by 483
  22. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  23. 2017Deep Layer Aggregationcited by 436
  24. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  25. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  26. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  27. 2020Are we done with ImageNet?cited by 329
  28. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  29. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  30. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  31. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  32. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  33. 2016What makes ImageNet good for transfer learning?cited by 192
  34. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  35. 2022MetaFormer Baselines for Visioncited by 131
  36. 2014Return of the Devil in the Details: Delving Deep into Convolutional Netscited by 122
  37. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  38. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 10; 2025: 12; 2026: 1.

35 papers.

Sort:

By area:

  1. 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 36
  2. 2025Do Multiple Instance Learning Models Transfer?cited by 28
  3. 2023Function-Space Regularization in Neural Networks: A Probabilistic Perspectivecited by 20
  4. 2023ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracycited by 19
  5. 2023ERM++: An Improved Baseline for Domain Generalizationcited by 18
  6. 2023Simplifying Neural Network Training Under Class Imbalancecited by 18
  7. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  8. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuningcited by 15
  9. 2023Does Progress On Object Recognition Benchmarks Improve Real-World Generalization?cited by 13
  10. 2024What Variables Affect Out-of-Distribution Generalization in Pretrained Models?cited by 13
  11. 2023Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representationscited by 11
  12. 2025Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learningcited by 11
23 more
  1. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  2. 2025Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culturecited by 10
  3. 2023Quantifying the Variability Collapse of Neural Networkscited by 9
  4. 2023A Holistic Assessment of the Reliability of Machine Learning Systemscited by 9
  5. 2023Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shiftscited by 6
  6. 2026When Priors Backfire: On the Vulnerability of Unlearnable Examples to Pretrainingcited by 5
  7. 2023GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shiftscited by 4
  8. 2025The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoningcited by 4
  9. 2023Automating Wood Species Detection and Classification in Microscopic Images of Fibrous Materials with Deep Learningcited by 3
  10. 2025Predictive Uncertainty for Runtime Assurance of a Real-Time Computer Vision-Based Landing Systemcited by 3
  11. 2024Challenging the Black Box: A Comprehensive Evaluation of Attribution Maps of CNN Applications in Agriculture and Forestrycited by 2
  12. 2024Vision Backbone Efficient Selection for Image Classification in Low-Data Regimescited by 2
  13. 2024Causality-Driven Audits of Model Robustnesscited by 2
  14. 2024Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurementcited by 2
  15. 2025Benchmarking Transferability: A Framework for Fair and Robust Evaluationcited by 2
  16. 2025Asymmetric Duos: Sidekicks Improve Uncertaintycited by 2
  17. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  18. 2025Tabular Data Adapters: Improving Outlier Detection for Unlabeled Private Datacited by 1
  19. 2023A Novel Site-Agnostic Multimodal Deep Learning Model to Identify Pro-Eating Disorder Content on Social Mediacited by 0
  20. 2024Optimizing for ROC Curves on Class-Imbalanced Data by Training over a Family of Loss Functionscited by 0
  21. 2024Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classificationcited by 0
  22. 2025Beyond ImageNet: Understanding Cross-Dataset Robustness of Lightweight Vision Modelscited by 0
  23. 2025BioBench: A Blueprint to Move Beyond ImageNet for Scientific ML Benchmarkscited by 0