# EXPANSE: A Deep Continual / Progressive Learning System for Deep Transfer Learning

arXiv: https://arxiv.org/abs/2205.10356 · HTML: https://mlsanity.com/paper/2205.10356/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 72 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 24 of 24)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [A Brief Review of Domain Adaptation](https://mlsanity.com/paper/2010.03978/) (2020)
- [Learning to Discover Novel Visual Categories via Deep Transfer Clustering](https://mlsanity.com/paper/1908.09884/) (2019)
- [Toward Understanding Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/1908.01091/) (2019)
- [Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types](https://mlsanity.com/paper/2103.13318/) (2021)
- [Growing a Brain: Fine-Tuning by Increasing Model Capacity](https://mlsanity.com/paper/1907.07844/) (2019)
- [Deep Transfer Learning for Multiple Class Novelty Detection](https://mlsanity.com/paper/1903.02196/) (2019)
- [Towards More Accurate Automatic Sleep Staging via Deep Transfer Learning](https://mlsanity.com/paper/1907.13177/) (2019)
- [Deep Transfer Learning for Person Re-identification](https://mlsanity.com/paper/1611.05244/) (2016)
- [Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks](https://mlsanity.com/paper/2002.09647/) (2020)
- [ImageNet pre-trained models with batch normalization](https://mlsanity.com/paper/1612.01452/) (2016)
- [Progressive Neural Networks for Transfer Learning in Emotion Recognition](https://mlsanity.com/paper/1706.03256/) (2017)
- [Deep Transfer Learning for Signal Detection in Ambient Backscatter Communications](https://mlsanity.com/paper/2009.05231/) (2020)
- [Understanding the Mechanisms of Deep Transfer Learning for Medical Images](https://mlsanity.com/paper/1704.06040/) (2017)
- [Deep Learning at the Edge](https://mlsanity.com/paper/1910.10231/) (2019)
- [Deep Transfer Learning for WiFi Localization](https://mlsanity.com/paper/2103.05123/) (2021)
- [Folding membrane proteins by deep transfer learning](https://mlsanity.com/paper/1708.08407/) (2017)

## Abstract (arXiv)

Deep transfer learning techniques try to tackle the limitations of deep learning, the dependency on extensive training data and the training costs, by reusing obtained knowledge. However, the current DTL techniques suffer from either catastrophic forgetting dilemma (losing the previously obtained knowledge) or overly biased pre-trained models (harder to adapt to target data) in finetuning pre-trained models or freezing a part of the pre-trained model, respectively. Progressive learning, a sub-category of DTL, reduces the effect of the overly biased model in the case of freezing earlier layers by adding a new layer to the end of a frozen pre-trained model. Even though it has been successful in many cases, it cannot yet handle distant source and target data. We propose a new continual/progressive learning approach for deep transfer learning to tackle these limitations. To avoid both catastrophic forgetting and overly biased-model problems, we expand the pre-trained model by expanding pre-trained layers (adding new nodes to each layer) in the model instead of only adding new layers. Hence the method is named EXPANSE. Our experimental results confirm that we can tackle distant source and target data using this technique. At the same time, the final model is still valid on the source data, achieving a promising deep continual learning approach. Moreover, we offer a new way of training deep learning models inspired by the human education system. We termed this two-step training: learning basics first, then adding complexities and uncertainties. The evaluation implies that the two-step training extracts more meaningful features and a finer basin on the error surface since it can achieve better accuracy in comparison to regular training. EXPANSE (model expansion and two-step training) is a systematic continual learning approach applicable to different problems and DL models.