How to Enhance Downstream Adversarial Robustness (almost) without Touching the Pre-Trained Foundation Model?
Not on the map. Why?
Abstract (arXiv)
With the rise of powerful foundation models, a pre-training-fine-tuning paradigm becomes increasingly popular these days: A foundation model is pre-trained using a huge amount of data from various sources, and then the downstream users only need to fine-tune and adapt it to specific downstream tasks. However, due to the high computation complexity of adversarial training, it is not feasible to fine-tune the foundation model to improve its robustness on the downstream task. Observing the above challenge, we want to improve the downstream robustness without updating/accessing the weights in the foundation model. Inspired from existing literature in robustness inheritance (Kim et al., 2020), through theoretical investigation, we identify a close relationship between robust contrastive learning with the adversarial robustness of supervised learning. To further validate and utilize this theoretical insight, we design a simple-yet-effective robust auto-encoder as a data pre-processing method before feeding the data into the foundation model. The proposed approach has zero access to the foundation model when training the robust auto-encoder. Extensive experiments demonstrate the effectiveness of the proposed method in improving the robustness of downstream tasks, verifying the connection between the feature robustness (implied by small adversarial contrastive loss) and the robustness of the downstream task.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 59 of its 71 references that are in the map. First posted on arXiv 15 Apr 2025.
Builds on
59 of 71 references.
References · 59
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2020Supervised Contrastive Learningcited by 2,829
- 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
- 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
- 2020What Makes for Good Views for Contrastive Learning?cited by 846
- 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
- 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
38 more
- 2020Overfitting in adversarially robust deep learningcited by 506
- 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
- 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
- 2020Debiased Contrastive Learningcited by 356
- 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
- 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
- 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
- 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 243
- 2021Improving Robustness using Generated Datacited by 234
- 2020What Should Not Be Contrastive in Contrastive Learningcited by 226
- 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
- 2017Towards Robust Neural Networks via Random Self-ensemblecited by 159
- 2020Adversarial Self-Supervised Contrastive Learningcited by 139
- 2018Rademacher Complexity for Adversarially Robust Generalizationcited by 132
- 2019Adversarial Training Can Hurt Generalizationcited by 132
- 2021On the Convergence and Robustness of Adversarial Trainingcited by 130
- 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
- 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
- 2018Curriculum Adversarial Trainingcited by 89
- 2020Precise Tradeoffs in Adversarial Training for Linear Regressioncited by 77
- 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 77
- 2019Adversarially robust transfer learningcited by 67
- 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 67
- 2022When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecturecited by 64
- 2017Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classificationcited by 56
- 2017Contrastive Learning for Image Captioningcited by 54
- 2020Precise Statistical Analysis of Classification Accuracies for Adversarial Trainingcited by 50
- 2019Robustness to Adversarial Perturbations in Learning from Incomplete Datacited by 49
- 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
- 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
- 2020The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalizationcited by 39
- 2020Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionalitycited by 36
- 2021Adversarial Training Helps Transfer Learning via Better Representationscited by 36
- 2022Stability Analysis and Generalization Bounds of Adversarial Trainingcited by 30
- 2020Sharp Statistical Guarantees for Adversarially Robust Gaussian Classificationcited by 28
- 2020More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Modelscited by 27
- 2020ARAE: Adversarially Robust Training of Autoencoders Improves Novelty Detectioncited by 27
- 2020Statistical Guarantees for Regularized Neural Networkscited by 25
50 most cited of 59 All 59 (JSON)
Built on by
Not cited yet.