mlsanityData to 14 Oct 2026

Adversarial Training Helps Transfer Learning via Better Representations

arXiv 2106.10189 ↗First posted 18 Jun 2021cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Transfer learning aims to leverage models pre-trained on source data to efficiently adapt to target setting, where only limited data are available for model fine-tuning. Recent works empirically demonstrate that adversarial training in the source data can improve the ability of models to transfer to new domains. However, why this happens is not known. In this paper, we provide a theoretical model to rigorously analyze how adversarial training helps transfer learning. We show that adversarial training in the source data generates provably better representations, so fine-tuning on top of this representation leads to a more accurate predictor of the target data. We further demonstrate both theoretically and empirically that semi-supervised learning in the source data can also improve transfer learning by similarly improving the representation. Moreover, performing adversarial training on top of semi-supervised learning can further improve transferability, suggesting that the two approaches have complementary benefits on representations. We support our theories with experiments on popular data sets and deep learning architectures.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 44 of its 71 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

44 of 71 references.

References · 44

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  8. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  9. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  10. 2016Adversarial Machine Learning at Scalecited by 936
  11. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  12. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
32 more
  1. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  2. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
  3. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  4. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  5. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 414
  6. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  7. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 368
  8. 2018Adversarially Robust Generalization Requires More Datacited by 318
  9. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  10. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
  11. 2018Certified Defenses against Adversarial Examplescited by 306
  12. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  13. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  14. 2016Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learningcited by 209
  15. 2015The Benefit of Multitask Representation Learningcited by 195
  16. 2020How Does Mixup Help With Robustness and Generalization?cited by 193
  17. 2016What makes ImageNet good for transfer learning?cited by 192
  18. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  19. 2019Are Labels Required for Improving Adversarial Robustness?cited by 163
  20. 2020Provable Meta-Learning of Linear Representationscited by 161
  21. 2020Certified Defenses for Adversarial Patchescited by 101
  22. 2019Instance adaptive adversarial training: Improved accuracy tradeoffs in neural netscited by 55
  23. 2020Improving Generalization in Meta-learning via Task Augmentationcited by 55
  24. 2021Meta-Learning with Fewer Tasks through Task Interpolationcited by 45
  25. 2015False Discoveries Occur Early on the Lasso Pathcited by 31
  26. 2020Improving Adversarial Robustness via Unlabeled Out-of-Domain Datacited by 23
  27. 2020Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classificationcited by 18
  28. 2018When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasetscited by 9
  29. 2020Interpreting Robust Optimization via Adversarial Influence Functionscited by 8
  30. 2021A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Modelscited by 8
  31. 2020Enhancing Certified Robustness via Smoothed Weighted Ensemblingcited by 7
  32. 2020Towards Understanding the Dynamics of the First-Order Adversariescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 12; 2023: 11; 2024: 7; 2025: 3; 2026: 1.

36 papers.

Sort:

By area:

  1. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  2. 2023Discover and Cure: Concept-aware Mitigation of Spurious Correlationcited by 71
  3. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  4. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  5. 2022Robustness Implies Generalization via Data-Dependent Generalization Boundscited by 23
  6. 2022Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noisecited by 22
  7. 2023HappyMap: A Generalized Multi-calibration Methodcited by 18
  8. 2023Adversarial Training of Self-supervised Monocular Depth Estimation against Physical-World Attackscited by 17
  9. 2023Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networkscited by 17
  10. 2022Towards Good Practices in Evaluating Transfer Adversarial Attackscited by 15
  11. 2022FIFA: Making Fairness More Generalizable in Classifiers Trained on Imbalanced Datacited by 13
  12. 2023Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learningcited by 11
24 more
  1. 2023NeRFool: Uncovering the Vulnerability of Generalizable Neural Radiance Fields against Adversarial Perturbationscited by 10
  2. 2022Reinforcement Learning with Stepwise Fairness Constraintscited by 9
  3. 2024Physics-Informed Neural Networks for High-Frequency and Multi-Scale Problems using Transfer Learningcited by 9
  4. 2021Scaffolding Setscited by 8
  5. 2022Adversarially Robust Models may not Transfer Better: Sufficient Conditions for Domain Transferability from the View of Regularizationcited by 8
  6. 2024Controlled Training Data Generation with Diffusion Modelscited by 8
  7. 2022MORA: Improving Ensemble Robustness Evaluation with Model-Reweighing Attackcited by 7
  8. 2022Generalizability of Adversarial Robustness Under Distribution Shiftscited by 6
  9. 2023SoK: Pitfalls in Evaluating Black-Box Attackscited by 6
  10. 2024Self-supervised Adversarial Training of Monocular Depth Estimation against Physical-World Attackscited by 5
  11. 2022Class-Level Logit Perturbationcited by 4
  12. 2022Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attackscited by 4
  13. 2025No Query, No Accesscited by 4
  14. 2022SHiFT: An Efficient, Flexible Search Engine for Transfer Learningcited by 3
  15. 2023Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learningcited by 3
  16. 2023On the Hardness of Robustness Transfer: A Perspective from Rademacher Complexity over Symmetric Difference Hypothesis Spacecited by 2
  17. 2024Theoretical Understanding of Learning from Adversarial Perturbationscited by 2
  18. 2023Is Certifying ℓp Robustness Still Worthwhile?cited by 1
  19. 2025On the Transfer of Knowledge in Quantum Algorithmscited by 1
  20. 2024Better Representations via Adversarial Training in Pre-Training: A Theoretical Perspectivecited by 0
  21. 2024Learning to Project for Cross-Task Knowledge Distillationcited by 0
  22. 2024SuDA: Support-based Domain Adaptation for Sim2Real Motion Capture with Flexible Sensorscited by 0
  23. 2025How to Enhance Downstream Adversarial Robustness (almost) without Touching the Pre-Trained Foundation Model?cited by 0
  24. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0