# Non-binary deep transfer learning for image classification

arXiv: https://arxiv.org/abs/2107.08585 · HTML: https://mlsanity.com/paper/2107.08585/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [Deep Face Recognition: A Survey](https://mlsanity.com/paper/1804.06655/) (2018)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning](https://mlsanity.com/paper/1806.06193/) (2018)
- [SpotTune: Transfer Learning through Adaptive Fine-tuning](https://mlsanity.com/paper/1811.08737/) (2018)
- [TResNet: High Performance GPU-Dedicated Architecture](https://mlsanity.com/paper/2003.13630/) (2020)
- [Learning Attentive Pairwise Interaction for Fine-Grained Classification](https://mlsanity.com/paper/2002.10191/) (2020)
- [DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks](https://mlsanity.com/paper/1901.09229/) (2019)
- [Rethinking the Hyperparameters for Fine-tuning](https://mlsanity.com/paper/2002.11770/) (2020)
- [Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuning](https://mlsanity.com/paper/1702.08690/) (2017)
- [DSOD: Learning Deeply Supervised Object Detectors from Scratch](https://mlsanity.com/paper/1708.01241/) (2017)
- [Deep learning in radiology: an overview of the concepts and a survey of the state of the art](https://mlsanity.com/paper/1802.08717/) (2018)
- [Domain Adaptive Transfer Learning with Specialist Models](https://mlsanity.com/paper/1811.07056/) (2018)
- [Analyzing the Performance of Multilayer Neural Networks for Object Recognition](https://mlsanity.com/paper/1407.1610/) (2014)
- [ScratchDet: Training Single-Shot Object Detectors from Scratch](https://mlsanity.com/paper/1810.08425/) (2018)
- [Towards Making Deep Transfer Learning Never Hurt](https://mlsanity.com/paper/1911.07489/) (2019)
- [Improving Object Detection from Scratch via Gated Feature Reuse](https://mlsanity.com/paper/1712.00886/) (2017)

## Built on by (most cited first, 2 of 2)

- [Deep transfer learning for image classification: a survey](https://mlsanity.com/paper/2205.09904/) (2022)
- [AMF: Adaptable Weighting Fusion with Multiple Fine-tuning for Image Classification](https://mlsanity.com/paper/2207.12944/) (2022)

## Abstract (arXiv)

The current standard for a variety of computer vision tasks using smaller numbers of labelled training examples is to fine-tune from weights pre-trained on a large image classification dataset such as ImageNet. The application of transfer learning and transfer learning methods tends to be rigidly binary. A model is either pre-trained or not pre-trained. Pre-training a model either increases performance or decreases it, the latter being defined as negative transfer. Application of L2-SP regularisation that decays the weights towards their pre-trained values is either applied or all weights are decayed towards 0. This paper re-examines these assumptions. Our recommendations are based on extensive empirical evaluation that demonstrate the application of a non-binary approach to achieve optimal results. (1) Achieving best performance on each individual dataset requires careful adjustment of various transfer learning hyperparameters not usually considered, including number of layers to transfer, different learning rates for different layers and different combinations of L2SP and L2 regularization. (2) Best practice can be achieved using a number of measures of how well the pre-trained weights fit the target dataset to guide optimal hyperparameters. We present methods for non-binary transfer learning including combining L2SP and L2 regularization and performing non-traditional fine-tuning hyperparameter searches. Finally we suggest heuristics for determining the optimal transfer learning hyperparameters. The benefits of using a non-binary approach are supported by final results that come close to or exceed state of the art performance on a variety of tasks that have traditionally been more difficult for transfer learning.