# Auto-Transfer: Learning to Route Transferrable Representations

arXiv: https://arxiv.org/abs/2202.01011 · HTML: https://mlsanity.com/paper/2202.01011/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 2 Feb 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [SpotTune: Transfer Learning through Adaptive Fine-tuning](https://mlsanity.com/paper/1811.08737/) (2018)
- [Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning](https://mlsanity.com/paper/1711.01239/) (2017)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)
- [DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks](https://mlsanity.com/paper/1901.09229/) (2019)
- [Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching](https://mlsanity.com/paper/2102.02973/) (2021)
- [Knowledge Transfer with Jacobian Matching](https://mlsanity.com/paper/1803.00443/) (2018)
- [Learning What and Where to Transfer](https://mlsanity.com/paper/1905.05901/) (2019)
- [The Unreasonable Effectiveness of Greedy Algorithms in Multi-Armed Bandit with Many Arms](https://mlsanity.com/paper/2002.10121/) (2020)
- [Supervised and Unsupervised Transfer Learning for Question Answering](https://mlsanity.com/paper/1711.05345/) (2017)
- [Improving Simple Models with Confidence Profiles](https://mlsanity.com/paper/1807.07506/) (2018)
- [Question Answering through Transfer Learning from Large Fine-grained Supervision Data](https://mlsanity.com/paper/1702.02171/) (2017)
- [MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and Architectures](https://mlsanity.com/paper/2006.07540/) (2020)

## Built on by (most cited first, 2 of 2)

- [Learngene: Inheriting Condensed Knowledge from the Ancestry Model to Descendant Models](https://mlsanity.com/paper/2305.02279/) (2023)
- [Weakly Supervised Learning with Automated Labels from Radiology Reports for Glioma Change Detection](https://mlsanity.com/paper/2210.09698/) (2022)

## Abstract (arXiv)

Knowledge transfer between heterogeneous source and target networks and tasks has received a lot of attention in recent times as large amounts of quality labeled data can be difficult to obtain in many applications. Existing approaches typically constrain the target deep neural network (DNN) feature representations to be close to the source DNNs feature representations, which can be limiting. We, in this paper, propose a novel adversarial multi-armed bandit approach that automatically learns to route source representations to appropriate target representations following which they are combined in meaningful ways to produce accurate target models. We see upwards of 5% accuracy improvements compared with the state-of-the-art knowledge transfer methods on four benchmark (target) image datasets CUB200, Stanford Dogs, MIT67, and Stanford40 where the source dataset is ImageNet. We qualitatively analyze the goodness of our transfer scheme by showing individual examples of the important features focused on by our target network at different layers compared with the (closest) competitors. We also observe that our improvement over other methods is higher for smaller target datasets making it an effective tool for small data applications that may benefit from transfer learning.