mlsanityData to 7 Oct 2026

When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

arXiv 2111.01124 ↗First posted 1 Nov 2021cs.CV cs.AI cs.LGcited by 76

Not on the map. Why?

Abstract (arXiv)

Contrastive learning (CL) can learn generalizable feature representations and achieve the state-of-the-art performance of downstream tasks by finetuning a linear classifier on top of it. However, as adversarial robustness becomes vital in image classification, it remains unclear whether or not CL is able to preserve robustness to downstream tasks. The main challenge is that in the self-supervised pretraining + supervised finetuning paradigm, adversarial robustness is easily forgotten due to a learning task mismatch from pretraining to finetuning. We call such a challenge 'cross-task robustness transferability'. To address the above problem, in this paper we revisit and advance CL principles through the lens of robustness enhancement. We show that (1) the design of contrastive views matters: High-frequency components of images are beneficial to improving model robustness; (2) Augmenting CL with pseudo-supervision stimulus (e.g., resorting to feature clustering) helps preserve robustness without forgetting. Equipped with our new designs, we propose AdvCL, a novel adversarial contrastive pretraining framework. We show that AdvCL is able to enhance cross-task robustness transferability without loss of model accuracy and finetuning efficiency. With a thorough experimental study, we demonstrate that AdvCL outperforms the state-of-the-art self-supervised robust learning methods across multiple datasets (CIFAR-10, CIFAR-100, and STL-10) and finetuning schemes (linear evaluation and full model finetuning).

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 54 of its 63 references that are in the map. First posted on arXiv 1 Nov 2021.

Builds on

54 of 63 references.

References · 54

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2020Supervised Contrastive Learningcited by 2,803
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  11. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  12. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
38 more
  1. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  2. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  3. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  5. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  6. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  7. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  8. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  9. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  10. 2019Adversarial Training for Free!cited by 638
  11. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  12. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  13. 2019Domain Generalization by Solving Jigsaw Puzzlescited by 449
  14. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  15. 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
  16. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  17. 2018Adversarial Logit Pairingcited by 224
  18. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  19. 2020Bag of Tricks for Adversarial Trainingcited by 199
  20. 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 181
  21. 2018Robustness via curvature regularization, and vice versacited by 174
  22. 2017EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examplescited by 154
  23. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 152
  24. 2019Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systemscited by 151
  25. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  26. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  27. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  28. 2019Defensive Quantization: When Efficiency Meets Robustnesscited by 95
  29. 2015Visualizing Deep Convolutional Neural Networks Using Natural Pre-Imagescited by 93
  30. 2019ClusterFit: Improving Generalization of Visual Representationscited by 82
  31. 2019Adversarially Robust Generalization Just Requires More Unlabeled Datacited by 75
  32. 2018Structured Adversarial Attack: Towards General Implementation and Better Interpretabilitycited by 63
  33. 2019Adversarial Objects Against LiDAR-Based Autonomous Driving Systemscited by 62
  34. 2018Training verified learners with learned verifierscited by 50
  35. 2019Robust Attribution Regularizationcited by 46
  36. 2019Selfie: Self-supervised Pretraining for Image Embeddingcited by 46
  37. 2019Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?cited by 42
  38. 2020Proper Network Interpretability Helps Adversarial Robustness in Classificationcited by 41

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 27; 2023: 23; 2024: 14; 2025: 5; 2026: 5.

76 papers.

Sort:

By area:

  1. 2022ADBench: Anomaly Detection Benchmarkcited by 230
  2. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  3. 2021Targeted Supervised Contrastive Learning for Long-Tailed Recognitioncited by 132
  4. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 119
  5. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  6. 2022Efficient and Effective Augmentation Strategy for Adversarial Trainingcited by 43
  7. 2022What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?cited by 39
  8. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  9. 2022Visual Prompting for Adversarial Robustnesscited by 33
  10. 2022Indiscriminate Poisoning Attacks on Unsupervised Contrastive Learningcited by 32
  11. 2022An Embarrassingly Simple Backdoor Attack on Self-supervised Learningcited by 30
  12. 2022Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustnesscited by 20
38 more
  1. 2023Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learningcited by 20
  2. 2023Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversationscited by 19
  3. 2023Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothingcited by 18
  4. 2022Reverse Engineering of Imperceptible Adversarial Image Perturbationscited by 17
  5. 2022Is Self-Supervised Learning More Robust Than Supervised Learning?cited by 17
  6. 2023Unlearnable Examples Give a False Sense of Security: Piercing through Unexploitable Data with Learnable Examplescited by 17
  7. 2022Pre-trained Adversarial Perturbationscited by 16
  8. 2022The Robustness Limits of SoTA Vision Models to Natural Variationcited by 15
  9. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  10. 2023Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selectioncited by 14
  11. 2022Adversarial Contrastive Learning via Asymmetric InfoNCEcited by 13
  12. 2024Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attackscited by 13
  13. 2022Understanding Adversarial Robustness Against On-manifold Adversarial Examplescited by 12
  14. 2023MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustnesscited by 12
  15. 2024Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Modelscited by 12
  16. 2021Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Frameworkcited by 11
  17. 2022Holistic Adversarial Robustness of Deep Learning Modelscited by 11
  18. 2023Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learningcited by 11
  19. 2023Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularizationcited by 10
  20. 2023Hard No-Box Adversarial Attack on Skeleton-Based Human Action Recognition with Skeleton-Motion-Informed Gradientcited by 10
  21. 2023Unveiling Vulnerabilities of Contrastive Recommender Systems to Poisoning Attackscited by 10
  22. 2022Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretrainingcited by 9
  23. 2023AutoLoRa: A Parameter-Free Automated Robust Fine-Tuning Frameworkcited by 9
  24. 2022Improving Adversarial Robustness by Contrastive Guided Diffusion Processcited by 8
  25. 2023On the Difficulty of Defending Contrastive Learning against Backdoor Attackscited by 8
  26. 2023Towards Robust Prompts on Vision-Language Modelscited by 7
  27. 2024AdvQDet: Detecting Query-Based Adversarial Attacks with Adversarial Contrastive Prompt Tuningcited by 7
  28. 2022Robust Fine-tuning via Perturbation and Interpolation from In-batch Instancescited by 6
  29. 2022Improving Adversarial Robustness with Self-Paced Hard-Class Pair Reweightingcited by 6
  30. 2022On Transfer of Adversarial Robustness from Pretraining to Downstream Taskscited by 5
  31. 2024Adversarial Feature Alignment: Balancing Robustness and Accuracy in Deep Learning via Adversarial Trainingcited by 5
  32. 2024Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Taskscited by 5
  33. 2022Adversarial Pretraining of Self-Supervised Deep Networks: Past, Present and Futurecited by 3
  34. 2023Robust Proxy: Improving Adversarial Robustness by Robust Proxy Learningcited by 3
  35. 2022SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Datacited by 2
  36. 2022Robustness of Unsupervised Representation Learning without Labelscited by 2
  37. 2022Effective Targeted Attacks for Adversarial Self-Supervised Learningcited by 2
  38. 2022CLAWSAT: Towards Both Robust and Accurate Code Modelscited by 2

50 most cited of 76 All 76 (JSON)