mlsanityData to 14 Oct 2026

Do Better ImageNet Models Transfer Better?

arXiv 1805.08974 ↗First posted 23 May 2018cs.CV cs.LG stat.MLcited by 565

Not on the map. Why?

Abstract (arXiv)

Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer. An implicit hypothesis in modern computer vision research is that models that perform better on ImageNet necessarily perform better on other vision tasks. However, this hypothesis has never been systematically tested. Here, we compare the performance of 16 classification networks on 12 image classification datasets. We find that, when networks are used as fixed feature extractors or fine-tuned, there is a strong correlation between ImageNet accuracy and transfer accuracy (r = 0.99 and 0.96, respectively). In the former setting, we find that this relationship is very sensitive to the way in which networks are trained on ImageNet; many common forms of regularization slightly improve ImageNet accuracy but yield penultimate layer features that are much worse for transfer learning. Additionally, we find that, on two small fine-grained image classification datasets, pretraining on ImageNet provides minimal benefits, indicating the learned features from ImageNet do not transfer well to fine-grained tasks. Together, our results show that ImageNet architectures generalize well across datasets, but ImageNet features are less general than previously suggested.

Key facts. Cited by 565 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 121; 2022: 117; 2023: 119; 2024: 85; 2025: 66; 2026: 57.

565 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  4. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  5. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  6. 2021Robust fine-tuning of zero-shot modelscited by 797
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  9. 2021ImageNet-21K Pretraining for the Massescited by 613
  10. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  11. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  12. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
38 more
  1. 2022Kubric: A scalable dataset generatorcited by 381
  2. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  3. 2021Explaining Neural Scaling Lawscited by 363
  4. 2022Visual Classification via Description from Large Language Modelscited by 336
  5. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  6. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  7. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
  8. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  9. 2021Projected GANs Converge Fastercited by 208
  10. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  11. 2021BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG datacited by 199
  12. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  13. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  14. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
  15. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 172
  16. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
  17. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 164
  18. 2022On-Device Training Under 256KB Memorycited by 160
  19. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  20. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
  21. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 150
  22. 2023Hyperbolic Image-Text Representationscited by 136
  23. 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
  24. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 118
  25. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  26. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  27. 2021Predicting with Confidence on Unseen Distributionscited by 116
  28. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  29. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 105
  30. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  31. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
  32. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  33. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  34. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  35. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 101
  36. 2021Pre-training without Natural Imagescited by 100
  37. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  38. 2021Parameter Prediction for Unseen Deep Architecturescited by 89

50 most cited of 565 All 565 (JSON)