mlsanityData to 7 Oct 2026

Generalization Bounds for Robust Contrastive Learning: From Theory to Practice

arXiv 2311.09671 ↗First posted 16 Nov 2023cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Contrastive Learning first extracts features from unlabeled data, followed by linear probing with labeled data. Adversarial Contrastive Learning (ACL) integrates Adversarial Training into the first phase to enhance feature robustness against attacks in the probing phase. While ACL has shown strong empirical results, its theoretical understanding remains limited. Furthermore, while a fair amount of theoretical works analyze how the unsupervised loss can support the supervised loss in the probing phase, none has examined its role to the robust supervised loss. To fill this gap, our work develops rigorous theories to identify which components in the unsupervised training can help improve the robust supervised loss. Specifically, besides the adversarial contrastive loss, we reveal that the benign one, along with a global divergence between benign and adversarial examples can also improve robustness. Proper experiments are conducted to justify our findings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

39 of 48 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2022Generative Adversarial Networkscited by 5,603
  8. 2013Intriguing properties of neural networkscited by 5,114
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
27 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  2. 2019Contrastive Multiview Codingcited by 1,297
  3. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  4. 2020Contrastive Learning with Hard Negative Samplescited by 619
  5. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  6. 2020Overfitting in adversarially robust deep learningcited by 504
  7. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  8. 2020Debiased Contrastive Learningcited by 354
  9. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  10. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  11. 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
  12. 2019Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Drivingcited by 151
  13. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 148
  14. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  15. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  16. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  17. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  18. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  19. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  20. 2020Contrastive Learning with Adversarial Examplescited by 91
  21. 2019Metric Learning for Adversarial Robustnesscited by 77
  22. 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 76
  23. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  24. 2023Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learningcited by 20
  25. 2020Improving Adversarial Robustness by Enforcing Local and Global Compactnesscited by 11
  26. 2023Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularizationcited by 10
  27. 2021Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learningcited by 8

Built on by

Not cited yet.