# Towards Understanding the Transferability of Deep Representations

arXiv: https://arxiv.org/abs/1909.12031 · HTML: https://mlsanity.com/paper/1909.12031/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2019.

## Built on by (most cited first, 12 of 12)

- [When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?](https://mlsanity.com/paper/2111.01124/) (2021)
- [Revisiting the Transferability of Supervised Pretraining: an MLP Perspective](https://mlsanity.com/paper/2112.00496/) (2021)
- [Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning](https://mlsanity.com/paper/2506.01339/) (2025)
- [Deep transfer learning for image classification: a survey](https://mlsanity.com/paper/2205.09904/) (2022)
- [Features are fate: a theory of transfer learning in high-dimensional regression](https://mlsanity.com/paper/2410.08194/) (2024)
- [Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning](https://mlsanity.com/paper/2310.08782/) (2023)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [CLAWSAT: Towards Both Robust and Accurate Code Models](https://mlsanity.com/paper/2211.11711/) (2022)
- [On the low-shot transferability of [V]-Mamba](https://mlsanity.com/paper/2403.10696/) (2024)
- [AMF: Adaptable Weighting Fusion with Multiple Fine-tuning for Image Classification](https://mlsanity.com/paper/2207.12944/) (2022)
- [CyclicFL: A Cyclic Model Pre-Training Approach to Efficient Federated Learning](https://mlsanity.com/paper/2301.12193/) (2023)
- [PMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Tasks](https://mlsanity.com/paper/2605.19145/) (2026)

## Abstract (arXiv)

Deep neural networks trained on a wide range of datasets demonstrate impressive transferability. Deep features appear general in that they are applicable to many datasets and tasks. Such property is in prevalent use in real-world applications. A neural network pretrained on large datasets, such as ImageNet, can significantly boost generalization and accelerate training if fine-tuned to a smaller target dataset. Despite its pervasiveness, few effort has been devoted to uncovering the reason of transferability in deep feature representations. This paper tries to understand transferability from the perspectives of improved generalization, optimization and the feasibility of transferability. We demonstrate that 1) Transferred models tend to find flatter minima, since their weight matrices stay close to the original flat region of pretrained parameters when transferred to a similar target dataset; 2) Transferred representations make the loss landscape more favorable with improved Lipschitzness, which accelerates and stabilizes training substantially. The improvement largely attributes to the fact that the principal component of gradient is suppressed in the pretrained parameters, thus stabilizing the magnitude of gradient in back-propagation. 3) The feasibility of transferability is related to the similarity of both input and label. And a surprising discovery is that the feasibility is also impacted by the training stages in that the transferability first increases during training, and then declines. We further provide a theoretical analysis to verify our observations.