mlsanityData to 7 Oct 2026

Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning

arXiv 2003.12862 ↗First posted 28 Mar 2020cs.CV cs.LGcited by 152

Not on the map. Why?

Abstract (arXiv)

Pretrained models from self-supervision are prevalently used in fine-tuning downstream tasks faster or for better accuracy. However, gaining robustness from pretraining is left unexplored. We introduce adversarial training into self-supervision, to provide general-purpose robust pre-trained models for the first time. We find these robust pre-trained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine-tuning. We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins (eg, 3.83% on robust accuracy and 1.3% on standard accuracy, on the CIFAR-10 dataset), compared with the conventional end-to-end adversarial training baseline. Moreover, we find that different self-supervised pre-trained models have a diverse adversarial vulnerability. It inspires us to ensemble several pretraining tasks, which boosts robustness more. Our ensemble strategy contributes to a further improvement of 3.59% on robust accuracy, while maintaining a slightly higher standard accuracy on CIFAR-10. Our codes are available at https://github.com/TAMU-VITA/Adv-SS-Pretraining.

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 45; 2022: 42; 2023: 30; 2024: 19; 2025: 11; 2026: 5.

152 papers.

Sort:

By area:

  1. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 246
  2. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
  3. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  4. 2021Proof Artifact Co-training for Theorem Proving with Language Modelscited by 130
  5. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  6. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  7. 2021Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attackscited by 101
  8. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 89
  9. 2023A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinkingcited by 85
  10. 2023Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Modelscited by 79
  11. 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 76
  12. 2021Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimizationcited by 75
38 more
  1. 2021Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networkscited by 73
  2. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  3. 2021MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaptioncited by 65
  4. 2021Probabilistic Margins for Instance Reweighting in Adversarial Trainingcited by 57
  5. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 55
  6. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
  7. 2021Online Adversarial Purification based on Self-Supervisioncited by 45
  8. 2024Current state of LLM Risks and AI Guardrailscited by 44
  9. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  10. 2021Are Vision Transformers Robust to Patch Perturbations?cited by 39
  11. 2022How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspectivecited by 39
  12. 2021Sinkhorn Distributionally Robust Optimizationcited by 38
  13. 2022Visual Prompting for Adversarial Robustnesscited by 33
  14. 2022DISCO: Adversarial Defense with Local Implicit Functionscited by 33
  15. 2024Revisiting Adversarial Training at Scalecited by 33
  16. 2021Attacking Adversarial Attacks as A Defensecited by 31
  17. 2021Low Curvature Activations Reduce Overfitting in Adversarial Trainingcited by 29
  18. 2021On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learningcited by 29
  19. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspectivecited by 28
  20. 2021Enhancing Adversarial Robustness via Test-time Transformation Ensemblingcited by 27
  21. 2021Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspectivecited by 25
  22. 2022Robust Models are less Over-Confidentcited by 25
  23. 2021LAFEAT: Piercing Through Adversarial Defenses with Latent Featurescited by 24
  24. 2021Adversarial Robustness against Multiple and Single lp-Threat Models via Quick Fine-Tuning of Robust Classifierscited by 23
  25. 2022Simplicial Embeddings in Self-Supervised Learning and Downstream Classificationcited by 23
  26. 2023Language-Driven Anchors for Zero-Shot Adversarial Robustnesscited by 23
  27. 2021Taxonomy of Machine Learning Safety: A Survey and Primercited by 22
  28. 2022TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimizationcited by 21
  29. 2022CNN Filter DB: An Empirical Investigation of Trained Convolutional Filterscited by 20
  30. 2022Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustnesscited by 20
  31. 2023Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothingcited by 18
  32. 2023TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalizationcited by 18
  33. 2022Reverse Engineering of Imperceptible Adversarial Image Perturbationscited by 17
  34. 2022Is Self-Supervised Learning More Robust Than Supervised Learning?cited by 17
  35. 2021Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learningcited by 16
  36. 2022Pre-trained Adversarial Perturbationscited by 16
  37. 2021How and When Adversarial Robustness Transfers in Knowledge Distillation?cited by 15
  38. 2022The Robustness Limits of SoTA Vision Models to Natural Variationcited by 15

50 most cited of 152 All 152 (JSON)