mlsanityData to 7 Oct 2026

Better Representations via Adversarial Training in Pre-Training: A Theoretical Perspective

arXiv 2401.15248 ↗First posted 26 Jan 2024cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pre-training is known to generate universal representations for downstream tasks in large-scale deep learning such as large language models. Existing literature, e.g., citekim2020adversarial, empirically observe that the downstream tasks can inherit the adversarial robustness of the pre-trained model. We provide theoretical justifications for this robustness inheritance phenomenon. Our theoretical results reveal that feature purification plays an important role in connecting the adversarial robustness of the pre-trained model and the downstream tasks in two-layer neural networks. Specifically, we show that (i) with adversarial training, each hidden node tends to pick only one (or a few) feature; (ii) without adversarial training, the hidden nodes can be vulnerable to attacks. This observation is valid for both supervised pre-training and contrastive learning. With purified nodes, it turns out that clean training is enough to achieve adversarial robustness in downstream tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 60 references that are in the map. First posted on arXiv 26 Jan 2024.

Builds on

49 of 60 references.

References · 49

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2020Supervised Contrastive Learningcited by 2,803
  6. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  7. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  8. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  9. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  10. 2020Overfitting in adversarially robust deep learningcited by 504
  11. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  12. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
37 more
  1. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  2. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 242
  3. 2021Improving Robustness using Generated Datacited by 233
  4. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  5. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  6. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  7. 2018Rademacher Complexity for Adversarially Robust Generalizationcited by 132
  8. 2019Adversarial Training Can Hurt Generalizationcited by 131
  9. 2021On the Convergence and Robustness of Adversarial Trainingcited by 127
  10. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  11. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  12. 2020Contrastive Learning with Adversarial Examplescited by 91
  13. 2018Curriculum Adversarial Trainingcited by 89
  14. 2020Precise Tradeoffs in Adversarial Training for Linear Regressioncited by 76
  15. 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 76
  16. 2019Adversarially robust transfer learningcited by 66
  17. 2022When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecturecited by 64
  18. 2019Robustness to Adversarial Perturbations in Learning from Incomplete Datacited by 49
  19. 2020Precise Statistical Analysis of Classification Accuracies for Adversarial Trainingcited by 49
  20. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  21. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 38
  22. 2020Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionalitycited by 36
  23. 2021Adversarial Training Helps Transfer Learning via Better Representationscited by 36
  24. 2022Stability Analysis and Generalization Bounds of Adversarial Trainingcited by 30
  25. 2020More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Modelscited by 27
  26. 2020Sharp Statistical Guarantees for Adversarially Robust Gaussian Classificationcited by 27
  27. 2020Statistical Guarantees for Regularized Neural Networkscited by 25
  28. 2020Improving Adversarial Robustness via Unlabeled Out-of-Domain Datacited by 23
  29. 2020On the Generalization Properties of Adversarial Trainingcited by 23
  30. 2022Adversarial Rademacher Complexity of Deep Neural Networkscited by 18
  31. 2022Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priorscited by 17
  32. 2022Adaptive Smoothness-weighted Adversarial Training for Multiple Perturbations with Its Stability Analysiscited by 9
  33. 2021CARTL: Cooperative Adversarially-Robust Transfer Learningcited by 8
  34. 2021Adversarial robustness for latent models: Revisiting the robust-standard accuracies tradeoffcited by 8
  35. 2022Blessing of Class Diversity in Pre-trainingcited by 4
  36. 2022Task-Agnostic Robust Representation Learningcited by 3
  37. 2022Robustness of Unsupervised Representation Learning without Labelscited by 2

Built on by

Not cited yet.