mlsanityData to 14 Oct 2026

On Transfer of Adversarial Robustness from Pretraining to Downstream Tasks

arXiv 2208.03835 ↗First posted 7 Aug 2022cs.LG cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

As large-scale training regimes have gained popularity, the use of pretrained models for downstream tasks has become common practice in machine learning. While pretraining has been shown to enhance the performance of models in practice, the transfer of robustness properties from pretraining to downstream tasks remains poorly understood. In this study, we demonstrate that the robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation, regardless of the protocol used for pretraining. We prove (i) a bound on the loss that holds independent of any downstream task, as well as (ii) a criterion for robust classification in particular. We validate our theoretical results in practical applications, show how our results can be used for calibrating expectations of downstream robustness, and when our results are useful for optimal transfer learning. Taken together, our results offer an initial step towards characterizing the requirements of the representation function for reliable post-adaptation performance.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 50 of its 62 references that are in the map. First posted on arXiv 7 Aug 2022.

Builds on

50 of 62 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2013Intriguing properties of neural networkscited by 5,155
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  12. 2016Wide Residual Networkscited by 3,070
38 more
  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  4. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  5. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  6. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  7. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  9. 2017Evasion Attacks against Machine Learning at Test Timecited by 689
  10. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  11. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
  12. 2018Do Better ImageNet Models Transfer Better?cited by 565
  13. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  14. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 330
  15. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
  16. 2018Certified Defenses against Adversarial Examplescited by 306
  17. 2021Unsolved Problems in ML Safetycited by 292
  18. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  19. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  20. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 185
  21. 2018Certified Adversarial Robustness with Additive Noisecited by 180
  22. 2018MMA Training: Direct Input Space Margin Maximization through Adversarial Trainingcited by 172
  23. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  24. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
  25. 2021A Universal Law of Robustness via Isoperimetrycited by 151
  26. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  27. 2020Adversarial Self-Supervised Contrastive Learningcited by 139
  28. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 114
  29. 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 77
  30. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 75
  31. 2019Adversarially robust transfer learningcited by 67
  32. 2021Improved Regularization and Robustness for Fine-tuning in Neural Networkscited by 45
  33. 2021How Fine-Tuning Allows for Effective Meta-Learningcited by 41
  34. 2019Certifiably Robust Interpretation in Deep Learningcited by 35
  35. 2022Does Robustness on ImageNet Transfer to Downstream Tasks?cited by 20
  36. 2021A Theoretical Analysis of Fine-tuning with Linear Teacherscited by 17
  37. 2022Adversarial vulnerability of powerful near out-of-distribution detectioncited by 16
  38. 2021ImageNet Pre-training also Transfers Non-Robustnesscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
  2. 2025Adapting to Evolving Adversaries with Regularized Continual Robust Trainingcited by 2
  3. 2025On the Interaction of Compressibility and Adversarial Robustnesscited by 2
  4. 2025Gradient Descent with Provably Tuned Learning-rate Schedulescited by 1
  5. 2025BIRD: Behavior Induction via Representation-structure Distillationcited by 0
  6. 2026Transferable Adversarial Robustness for Speech Foundation Models via Hierarchical Stabilizationcited by 0