mlsanityData to 14 Oct 2026

How transferable are features in deep neural networks?

arXiv 1411.1792 ↗First posted 6 Nov 2014cs.LG cs.NEcited by 1,225

Not on the map. Why?

Abstract (arXiv)

Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.

Key facts. Cited by 1,225 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Nov 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 2; 2019: 0; 2020: 0; 2021: 282; 2022: 227; 2023: 208; 2024: 168; 2025: 186; 2026: 152.

1,225 papers.

Sort:

By area:

  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  2. 2022Visual Prompt Tuningcited by 1,681
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  4. 2021Domain Generalization: A Surveycited by 764
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  6. 2021Extract Free Dense Labels from CLIPcited by 583
  7. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  8. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  9. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  11. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  12. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
38 more
  1. 2022Matryoshka Representation Learningcited by 282
  2. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  3. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  4. 2023A Cookbook of Self-Supervised Learningcited by 250
  5. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  6. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  7. 2023Glaze: Protecting Artists from Style Mimicry by Text-to-Image Modelscited by 205
  8. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  9. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
  10. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  11. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
  12. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 159
  13. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 154
  14. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  15. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  16. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  17. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  18. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 131
  19. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 126
  20. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 122
  21. 2021Cycle Self-Training for Domain Adaptationcited by 120
  22. 2021Deep Subdomain Adaptation Network for Image Classificationcited by 119
  23. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  24. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  25. 2023ChipGPT: How far are we from natural language hardware designcited by 105
  26. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  27. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  28. 2021Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognitioncited by 95
  29. 2022Three things everyone should know about Vision Transformerscited by 90
  30. 2021Ensembling Off-the-shelf Models for GAN Trainingcited by 86
  31. 2024Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classificationcited by 85
  32. 2021Proof-of-Learning: Definitions and Practicecited by 84
  33. 2021Deep Learning for Chest X-ray Analysis: A Surveycited by 84
  34. 2022Deep transfer operator learning for partial differential equations under conditional shiftcited by 83
  35. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 82
  36. 2021TVT: Transferable Vision Transformer for Unsupervised Domain Adaptationcited by 81
  37. 2022An Information-Theoretic Approach to Transferability in Task Transfer Learningcited by 81
  38. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80

50 most cited of 1,225 All 1,225 (JSON)