# Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classification

arXiv: https://arxiv.org/abs/2007.05869 · HTML: https://mlsanity.com/paper/2007.05869/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2020.

## Built on by (most cited first, 18 of 18)

- [Improved OOD Generalization via Adversarial Training and Pre-training](https://mlsanity.com/paper/2105.11144/) (2021)
- [Adversarial Training Helps Transfer Learning via Better Representations](https://mlsanity.com/paper/2106.10189/) (2021)
- [A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks](https://mlsanity.com/paper/2106.02105/) (2021)
- [A Data-Based Perspective on Transfer Learning](https://mlsanity.com/paper/2207.05739/) (2022)
- [On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learning](https://mlsanity.com/paper/2102.10454/) (2021)
- [Noisy Feature Mixup](https://mlsanity.com/paper/2110.02180/) (2021)
- [When does Bias Transfer in Transfer Learning?](https://mlsanity.com/paper/2207.02842/) (2022)
- [Adversarial Training is Not Ready for Robot Learning](https://mlsanity.com/paper/2103.08187/) (2021)
- [Unsupervised Robust Domain Adaptation without Source Data](https://mlsanity.com/paper/2103.14577/) (2021)
- [Combining Diverse Feature Priors](https://mlsanity.com/paper/2110.08220/) (2021)
- [Non-generative Generalized Zero-shot Learning via Task-correlated Disentanglement and Controllable Samples Synthesis](https://mlsanity.com/paper/2203.05335/) (2022)
- [Adversarial Perturbations Are Not So Weird: Entanglement of Robust and Non-Robust Features in Neural Network Classifiers](https://mlsanity.com/paper/2102.05110/) (2021)
- [Domain Invariant Adversarial Learning](https://mlsanity.com/paper/2104.00322/) (2021)
- [Improving filling level classification with adversarial training](https://mlsanity.com/paper/2102.04057/) (2021)
- [Learning from learning machines: a new generation of AI technology to meet the needs of science](https://mlsanity.com/paper/2111.13786/) (2021)
- [Adversarially trained neural representations may already be as robust as corresponding biological neural representations](https://mlsanity.com/paper/2206.11228/) (2022)
- [On Generalization of Graph Autoencoders with Adversarial Training](https://mlsanity.com/paper/2107.02658/) (2021)
- [ARIA: Adversarially Robust Image Attribution for Content Provenance](https://mlsanity.com/paper/2202.12860/) (2022)

## Abstract (arXiv)

Transfer learning has emerged as a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains. This process consists of taking a neural network pre-trained on a large feature-rich source dataset, freezing the early layers that encode essential generic image properties, and then fine-tuning the last few layers in order to capture specific information related to the target situation. This approach is particularly useful when only limited or weakly labeled data are available for the new task. In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models, especially if only limited data are available for the new domain task. Further, we observe that adversarial training biases the learnt representations to retaining shapes, as opposed to textures, which impacts the transferability of the source models. Finally, through the lens of influence functions, we discover that transferred adversarially-trained models contain more human-identifiable semantic information, which explains -- at least partly -- why adversarially-trained models transfer better.