# Does progress on ImageNet transfer to real-world datasets?

arXiv: https://arxiv.org/abs/2301.04644 · HTML: https://mlsanity.com/paper/2301.04644/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 87 references that are in the map.
- First posted on arXiv 11 Jan 2023.

## Builds on (most cited first, 30 of 59)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)

## Built on by (most cited first, 30 of 35)

- [Medical Large Language Model Benchmarks Should Prioritize Construct Validity](https://mlsanity.com/paper/2503.10694/) (2025)
- [Do Multiple Instance Learning Models Transfer?](https://mlsanity.com/paper/2506.09022/) (2025)
- [Function-Space Regularization in Neural Networks: A Probabilistic Perspective](https://mlsanity.com/paper/2312.17162/) (2023)
- [ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy](https://mlsanity.com/paper/2311.09215/) (2023)
- [ERM++: An Improved Baseline for Domain Generalization](https://mlsanity.com/paper/2304.01973/) (2023)
- [Simplifying Neural Network Training Under Class Imbalance](https://mlsanity.com/paper/2312.02517/) (2023)
- [On Catastrophic Inheritance of Large Foundation Models](https://mlsanity.com/paper/2402.01909/) (2024)
- [Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning](https://mlsanity.com/paper/2406.11148/) (2024)
- [Does Progress On Object Recognition Benchmarks Improve Real-World Generalization?](https://mlsanity.com/paper/2307.13136/) (2023)
- [What Variables Affect Out-of-Distribution Generalization in Pretrained Models?](https://mlsanity.com/paper/2405.15018/) (2024)
- [Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations](https://mlsanity.com/paper/2306.08658/) (2023)
- [Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning](https://mlsanity.com/paper/2502.10691/) (2025)
- [Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culture](https://mlsanity.com/paper/2502.15620/) (2025)
- [Efficient Lifelong Model Evaluation in an Era of Rapid Progress](https://mlsanity.com/paper/2402.19472/) (2024)
- [Quantifying the Variability Collapse of Neural Networks](https://mlsanity.com/paper/2306.03440/) (2023)
- [A Holistic Assessment of the Reliability of Machine Learning Systems](https://mlsanity.com/paper/2307.10586/) (2023)
- [Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shifts](https://mlsanity.com/paper/2310.13040/) (2023)
- [When Priors Backfire: On the Vulnerability of Unlearnable Examples to Pretraining](https://mlsanity.com/paper/2603.04731/) (2026)
- [GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts](https://mlsanity.com/paper/2310.08677/) (2023)
- [The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoning](https://mlsanity.com/paper/2510.04141/) (2025)
- [Automating Wood Species Detection and Classification in Microscopic Images of Fibrous Materials with Deep Learning](https://mlsanity.com/paper/2307.09588/) (2023)
- [Asymmetric Duos: Sidekicks Improve Uncertainty](https://mlsanity.com/paper/2505.18636/) (2025)
- [Predictive Uncertainty for Runtime Assurance of a Real-Time Computer Vision-Based Landing System](https://mlsanity.com/paper/2508.09732/) (2025)
- [Data-Centric Lessons To Improve Speech-Language Pretraining](https://mlsanity.com/paper/2510.20860/) (2025)
- [Challenging the Black Box: A Comprehensive Evaluation of Attribution Maps of CNN Applications in Agriculture and Forestry](https://mlsanity.com/paper/2402.11670/) (2024)
- [Vision Backbone Efficient Selection for Image Classification in Low-Data Regimes](https://mlsanity.com/paper/2410.08592/) (2024)
- [Causality-Driven Audits of Model Robustness](https://mlsanity.com/paper/2410.23494/) (2024)
- [Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurement](https://mlsanity.com/paper/2411.01099/) (2024)
- [Benchmarking Transferability: A Framework for Fair and Robust Evaluation](https://mlsanity.com/paper/2504.20121/) (2025)
- [Tabular Data Adapters: Improving Outlier Detection for Unlabeled Private Data](https://mlsanity.com/paper/2504.20862/) (2025)

## Abstract (arXiv)

Does progress on ImageNet transfer to real-world datasets? We investigate this question by evaluating ImageNet pre-trained models with varying accuracy (57% - 83%) on six practical image classification datasets. In particular, we study datasets collected with the goal of solving real-world tasks (e.g., classifying images from camera traps or satellites), as opposed to web-scraped benchmarks collected for comparing models. On multiple datasets, models with higher ImageNet accuracy do not consistently yield performance improvements. For certain tasks, interventions such as data augmentation improve performance even when architectures do not. We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.