# A Review of Deep Transfer Learning and Recent Advancements

arXiv: https://arxiv.org/abs/2201.09679 · HTML: https://mlsanity.com/paper/2201.09679/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 72 references that are in the map.
- First posted on arXiv 19 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [A Brief Review of Domain Adaptation](https://mlsanity.com/paper/2010.03978/) (2020)
- [Learning to Discover Novel Visual Categories via Deep Transfer Clustering](https://mlsanity.com/paper/1908.09884/) (2019)
- [Toward Understanding Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/1908.01091/) (2019)
- [Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types](https://mlsanity.com/paper/2103.13318/) (2021)
- [Growing a Brain: Fine-Tuning by Increasing Model Capacity](https://mlsanity.com/paper/1907.07844/) (2019)
- [Deep Transfer Learning for Multiple Class Novelty Detection](https://mlsanity.com/paper/1903.02196/) (2019)
- [Towards More Accurate Automatic Sleep Staging via Deep Transfer Learning](https://mlsanity.com/paper/1907.13177/) (2019)
- [Deep Transfer Learning for Person Re-identification](https://mlsanity.com/paper/1611.05244/) (2016)
- [Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks](https://mlsanity.com/paper/2002.09647/) (2020)
- [ImageNet pre-trained models with batch normalization](https://mlsanity.com/paper/1612.01452/) (2016)
- [Progressive Neural Networks for Transfer Learning in Emotion Recognition](https://mlsanity.com/paper/1706.03256/) (2017)
- [Deep Transfer Learning for Signal Detection in Ambient Backscatter Communications](https://mlsanity.com/paper/2009.05231/) (2020)
- [Understanding the Mechanisms of Deep Transfer Learning for Medical Images](https://mlsanity.com/paper/1704.06040/) (2017)
- [Deep Learning at the Edge](https://mlsanity.com/paper/1910.10231/) (2019)
- [Deep Transfer Learning for WiFi Localization](https://mlsanity.com/paper/2103.05123/) (2021)
- [Folding membrane proteins by deep transfer learning](https://mlsanity.com/paper/1708.08407/) (2017)

## Built on by (most cited first, 30 of 57)

- [Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing](https://mlsanity.com/paper/2310.06234/) (2023)
- [Learngene: Inheriting Condensed Knowledge from the Ancestry Model to Descendant Models](https://mlsanity.com/paper/2305.02279/) (2023)
- [Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach](https://mlsanity.com/paper/2403.19067/) (2024)
- [A Survey on Deep Tabular Learning](https://mlsanity.com/paper/2410.12034/) (2024)
- [Deep transfer learning for image classification: a survey](https://mlsanity.com/paper/2205.09904/) (2022)
- [CarDreamer: Open-Source Learning Platform for World Model based Autonomous Driving](https://mlsanity.com/paper/2405.09111/) (2024)
- [Reconstructing Training Data From Real World Models Trained with Transfer Learning](https://mlsanity.com/paper/2407.15845/) (2024)
- [Transferring Core Knowledge via Learngenes](https://mlsanity.com/paper/2401.08139/) (2024)
- [Genes in Intelligent Agents](https://mlsanity.com/paper/2306.10225/) (2023)
- [A Recent Survey of Heterogeneous Transfer Learning](https://mlsanity.com/paper/2310.08459/) (2023)
- [Bridging Data Gaps in Healthcare: A Scoping Review of Transfer Learning in Biomedical Data Analysis](https://mlsanity.com/paper/2407.11034/) (2024)
- [Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation](https://mlsanity.com/paper/2410.22952/) (2024)
- [Transfer learning for day-ahead load forecasting: a case study on European national electricity demand time series](https://mlsanity.com/paper/2310.15555/) (2023)
- [How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey](https://mlsanity.com/paper/2312.04775/) (2023)
- [Adapting WavLM for Speech Emotion Recognition](https://mlsanity.com/paper/2405.04485/) (2024)
- [Can we learn better with hard samples?](https://mlsanity.com/paper/2304.03486/) (2023)
- [Transfer learning with generative models for object detection on limited datasets](https://mlsanity.com/paper/2402.06784/) (2024)
- [Computer Vision in the Food Industry: Accurate, Real-time, and Automatic Food Recognition with Pretrained MobileNetV2](https://mlsanity.com/paper/2405.11621/) (2024)
- [Federated and Transfer Learning for Cancer Detection Based on Image Analysis](https://mlsanity.com/paper/2405.20126/) (2024)
- [In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks](https://mlsanity.com/paper/2410.01548/) (2024)
- [What do physics-informed DeepONets learn? Understanding and improving training for scientific computing applications](https://mlsanity.com/paper/2411.18459/) (2024)
- [Implicit regularisation in stochastic gradient descent: from single-objective to two-player games](https://mlsanity.com/paper/2307.05789/) (2023)
- [Hyperdimensional computing: a fast, robust and interpretable paradigm for biological data](https://mlsanity.com/paper/2402.17572/) (2024)
- [Colab NAS: Obtaining lightweight task-specific convolutional neural networks following Occam's razor](https://mlsanity.com/paper/2212.07700/) (2022)
- [Mobile Traffic Prediction at the Edge Through Distributed and Deep Transfer Learning](https://mlsanity.com/paper/2310.14456/) (2023)
- [An Integrated Imitation and Reinforcement Learning Methodology for Robust Agile Aircraft Control with Limited Pilot Demonstration Data](https://mlsanity.com/paper/2401.08663/) (2023)
- [Online Transfer Learning for RSV Case Detection](https://mlsanity.com/paper/2402.01987/) (2024)
- [Text2Data: Low-Resource Data Generation with Textual Control](https://mlsanity.com/paper/2402.10941/) (2024)
- [Bias correction of wind power forecasts with SCADA data and continuous learning](https://mlsanity.com/paper/2402.13916/) (2024)
- [Data-driven Modeling in Metrology -- A Short Introduction, Current Developments and Future Perspectives](https://mlsanity.com/paper/2406.16659/) (2024)

## Abstract (arXiv)

Deep learning has been the answer to many machine learning problems during the past two decades. However, it comes with two major constraints: dependency on extensive labeled data and training costs. Transfer learning in deep learning, known as Deep Transfer Learning (DTL), attempts to reduce such dependency and costs by reusing an obtained knowledge from a source data/task in training on a target data/task. Most applied DTL techniques are network/model-based approaches. These methods reduce the dependency of deep learning models on extensive training data and drastically decrease training costs. As a result, researchers detected Covid-19 infection on chest X-Rays with high accuracy at the beginning of the pandemic with minimal data using DTL techniques. Also, the training cost reduction makes DTL viable on edge devices with limited resources. Like any new advancement, DTL methods have their own limitations, and a successful transfer depends on some adjustments for different scenarios. In this paper, we review the definition and taxonomy of deep transfer learning and well-known methods. Then we investigate the DTL approaches by reviewing recent applied DTL techniques in the past five years. Further, we review some experimental analyses of DTLs to learn the best practice for applying DTL in different scenarios. Moreover, the limitations of DTLs (catastrophic forgetting dilemma and overly biased pre-trained models) are discussed, along with possible solutions and research trends.