mlsanityData to 14 Oct 2026

ProFeAT: Projected Feature Adversarial Training for Self-Supervised Learning of Robust Representations

arXiv 2406.05796 ↗First posted 9 Jun 2024cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

The need for abundant labelled data in supervised Adversarial Training (AT) has prompted the use of Self-Supervised Learning (SSL) techniques with AT. However, the direct application of existing SSL methods to adversarial training has been sub-optimal due to the increased training complexity of combining SSL with AT. A recent approach, DeACL, mitigates this by utilizing supervision from a standard SSL teacher in a distillation setting, to mimic supervised AT. However, we find that there is still a large performance gap when compared to supervised adversarial training, specifically on larger models. In this work, investigate the key reason for this gap and propose Projected Feature Adversarial Training (ProFeAT) to bridge the same. We show that the sub-optimal distillation performance is a result of mismatch in training objectives of the teacher and student, and propose to use a projection head at the student, that allows it to leverage weak supervision from the teacher while also being able to learn adversarially robust representations that are distinct from the teacher. We further propose appropriate attack and defense losses at the feature and projector, alongside a combination of weak and strong augmentations for the teacher and student respectively, to improve the training data diversity without increasing the training complexity. Through extensive experiments on several benchmark datasets and models, we demonstrate significant improvements in both clean and robust accuracy when compared to existing SSL-AT methods, setting a new state-of-the-art. We further report on-par/ improved performance when compared to TRADES, a popular supervised-AT method.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 58 references that are in the map. First posted on arXiv 9 Jun 2024.

Builds on

49 of 58 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2013Intriguing properties of neural networkscited by 5,155
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2016Wide Residual Networkscited by 3,070
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
37 more
  1. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  2. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  3. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  4. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  5. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  6. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  7. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  8. 2017Ensemble Adversarial Training: Attacks and Defensescited by 840
  9. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  10. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  11. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
  12. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  13. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  14. 2020Overfitting in adversarially robust deep learningcited by 506
  15. 2019On Evaluating Adversarial Robustnesscited by 447
  16. 2017Mitigating Adversarial Effects Through Randomizationcited by 383
  17. 2018Adversarially Robust Generalization Requires More Datacited by 318
  18. 2018Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionalitycited by 282
  19. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
  20. 2017PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examplescited by 244
  21. 2020Bag of Tricks for Adversarial Trainingcited by 200
  22. 2018Stochastic Activation Pruning for Robust Adversarial Defensecited by 171
  23. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
  24. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  25. 2020Adversarial Self-Supervised Contrastive Learningcited by 139
  26. 2020RayS: A Ray Searching Method for Hard-label Adversarial Attackcited by 96
  27. 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 77
  28. 2020Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defensescited by 62
  29. 2020Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbationscited by 55
  30. 2019Selfie: Self-supervised Pretraining for Image Embeddingcited by 46
  31. 2022Efficient and Effective Augmentation Strategy for Adversarial Trainingcited by 43
  32. 2021DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learningcited by 26
  33. 2022Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learningcited by 26
  34. 2022SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillationcited by 21
  35. 2022Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustnesscited by 20
  36. 2023Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learningcited by 20
  37. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20

Built on by

Not cited yet.