ImageNet Pre-training also Transfers Non-Robustness
Not on the map. Why?
Abstract (arXiv)
ImageNet pre-training has enabled state-of-the-art results on many tasks. In spite of its recognized contribution to generalization, we observed in this study that ImageNet pre-training also transfers adversarial non-robustness from pre-trained model into fine-tuned model in the downstream classification tasks. We first conducted experiments on various datasets and network backbones to uncover the adversarial non-robustness in fine-tuned model. Further analysis was conducted on examining the learned knowledge of fine-tuned model and standard model, and revealed that the reason leading to the non-robustness is the non-robust features transferred from ImageNet pre-trained model. Finally, we analyzed the preference for feature learning of the pre-trained model, explored the factors influencing robustness, and introduced a simple robust ImageNet pre-training solution. Our code is available at urlhttps://github.com/jiamingzhang94/ImageNet-Pretraining-transfers-non-robustness.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 21 Jun 2021.
Builds on
31 of 46 references.
References · 31
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
- 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
- 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
- 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
- 2014How transferable are features in deep neural networks?cited by 1,225
- 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
- 2016Adversarial Machine Learning at Scalecited by 936
19 more
- 2016Universal adversarial perturbationscited by 784
- 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
- 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
- 2018Rethinking ImageNet Pre-trainingcited by 377
- 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
- 2020Rethinking Pre-training and Self-trainingcited by 352
- 2020Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networkscited by 349
- 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
- 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
- 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 288
- 2019Billion-scale semi-supervised learning for image classificationcited by 232
- 2017Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?cited by 221
- 2017Generative Adversarial Perturbationscited by 153
- 2019Self-Training for End-to-End Speech Recognitioncited by 129
- 2019Adversarially robust transfer learningcited by 67
- 2018Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representationcited by 62
- 2019Adversarial Robustness as a Prior for Learned Representationscited by 20
- 2019Knowledge Consistency between Neural Networks and Beyondcited by 16
- 2020Adversarial Privacy-preserving Filtercited by 13
Built on by
Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1; 2026: 1.
3 papers.