RUSH: Robust Contrastive Learning via Randomized Smoothing
Not on the map. Why?
Abstract (arXiv)
Recently, adversarial training has been incorporated in self-supervised contrastive pre-training to augment label efficiency with exciting adversarial robustness. However, the robustness came at a cost of expensive adversarial training. In this paper, we show a surprising fact that contrastive pre-training has an interesting yet implicit connection with robustness, and such natural robustness in the pre trained representation enables us to design a powerful robust algorithm against adversarial attacks, RUSH, that combines the standard contrastive pre-training and randomized smoothing. It boosts both standard accuracy and robust accuracy, and significantly reduces training costs as compared with adversarial training. We use extensive empirical studies to show that the proposed RUSH outperforms robust classifiers from adversarial training, by a significant margin on common benchmarks (CIFAR-10, CIFAR-100, and STL-10) under first-order attacks. In particular, under ℓ∞-norm perturbations of size 8/255 PGD attack on CIFAR-10, our model using ResNet-18 as backbone reached 77.8% robust accuracy and 87.9% standard accuracy. Our work has an improvement of over 15% in robust accuracy and a slight improvement in standard accuracy, compared to the state-of-the-arts.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 11 Jul 2022.
Builds on
39 of 48 references.
References · 39
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
- 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
- 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
- 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
- 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
- 2016Adversarial Machine Learning at Scalecited by 936
- 2018Robustness May Be at Odds with Accuracycited by 880
27 more
- 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 744
- 2020Fast is better than free: Revisiting adversarial trainingcited by 727
- 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
- 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
- 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 657
- 2019Adversarial Training for Free!cited by 643
- 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
- 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
- 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
- 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 460
- 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 330
- 2018Towards Fast Computation of Certified Robustness for ReLU Networkscited by 255
- 2018Adversarial Logit Pairingcited by 224
- 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 204
- 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 183
- 2018Semidefinite relaxations for certifying robustness to adversarial examplescited by 179
- 2019Are Labels Required for Improving Adversarial Robustness?cited by 163
- 2017Towards Robust Neural Networks via Random Self-ensemblecited by 159
- 2020Torchattacks: A PyTorch Repository for Adversarial Attackscited by 158
- 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
- 2020Randomized Smoothing of All Shapes and Sizescited by 147
- 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
- 2020Adversarial Self-Supervised Contrastive Learningcited by 139
- 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 77
- 2019Robustness Certificates for Sparse Adversarial Attacks by Randomized Ablationcited by 60
- 2017Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classificationcited by 56
- 2020Certifying Confidence via Randomized Smoothingcited by 36
Built on by
Citing papers per year: 2022: 1.
1 paper.