mlsanityData to 14 Oct 2026

Is Self-Supervised Learning More Robust Than Supervised Learning?

arXiv 2206.05259 ↗First posted 10 Jun 2022cs.CV cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Self-supervised contrastive learning is a powerful tool to learn visual representation without labels. Prior work has primarily focused on evaluating the recognition accuracy of various pre-training algorithms, but has overlooked other behavioral aspects. In addition to accuracy, distributional robustness plays a critical role in the reliability of machine learning models. We design and conduct a series of robustness tests to quantify the behavioral differences between contrastive learning and supervised learning to downstream or pre-training data distribution changes. These tests leverage data corruptions at multiple levels, ranging from pixel-level gamma distortion to patch-level shuffling and to dataset-level distribution shift. Our tests unveil intriguing robustness behaviors of contrastive and supervised learning. On the one hand, under downstream corruptions, we generally observe that contrastive learning is surprisingly more robust than supervised learning. On the other hand, under pre-training corruptions, we find contrastive learning vulnerable to patch shuffling and pixel intensity change, yet less sensitive to dataset-level distribution change. We attempt to explain these results through the role of data augmentation and feature space properties. Our insight has implications in improving the downstream robustness of supervised learning.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 61 of its 76 references that are in the map. First posted on arXiv 10 Jun 2022.

Builds on

61 of 76 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
38 more
  1. 2013Intriguing properties of neural networkscited by 5,155
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  8. 2020Supervised Contrastive Learningcited by 2,829
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  13. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  14. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  15. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  16. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  17. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  18. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  19. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  20. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  21. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  22. 2019Adversarial Training for Free!cited by 643
  23. 2017Large Batch Training of Convolutional Networkscited by 535
  24. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  25. 2020Understanding the Behaviour of Contrastive Losscited by 398
  26. 2020What is being transferred in transfer learning?cited by 368
  27. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
  28. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  29. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 198
  30. 2020A Self-supervised Approach for Adversarial Robustnesscited by 180
  31. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
  32. 2020Adversarial Self-Supervised Contrastive Learningcited by 139
  33. 2020Intriguing Properties of Contrastive Lossescited by 130
  34. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 123
  35. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  36. 2020What makes instance discrimination good for transfer learning?cited by 111
  37. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
  38. 2021Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentationcited by 101

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 7; 2024: 4; 2025: 2.

17 papers.

Sort:

By area:

  1. 2023Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Datacited by 44
  2. 2023Continuous Learning for Android Malware Detectioncited by 33
  3. 2022An Embarrassingly Simple Backdoor Attack on Self-supervised Learningcited by 31
  4. 2023Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labelscited by 20
  5. 2022Cyclical Self-Supervision for Semi-Supervised Ejection Fraction Prediction from Echocardiogram Videoscited by 12
  6. 2024A Unified Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capabilitycited by 9
  7. 2022A picture of the space of typical learnable taskscited by 7
  8. 2022Self-Supervised Learning with Limited Labeled Data for Prostate Cancer Detection in High Frequency Ultrasoundcited by 7
  9. 2023Understanding Self-Supervised Pretraining with Part-Aware Representation Learningcited by 7
  10. 2024How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrievalcited by 6
  11. 2023Can Pre-trained Networks Detect Familiar Out-of-Distribution Data?cited by 4
  12. 2024Understanding Representation Learnability of Nonlinear Self-Supervised Learningcited by 2
5 more
  1. 2024Evaluating Self-Supervised Learning in Medical Imaging: A Benchmark for Robustness, Generalizability, and Multi-Domain Impactcited by 2
  2. 2025Adversarial Robustness of Discriminative Self-Supervised Learning in Visioncited by 1
  3. 2025A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capabilitycited by 1
  4. 2023Rethinking Robust Contrastive Learning from the Adversarial Perspectivecited by 0
  5. 2023Variational Self-Supervised Contrastive Learning Using Beta Divergencecited by 0