mlsanityData to 7 Oct 2026

A Study of the Generalizability of Self-Supervised Representations

arXiv 2109.09150 ↗First posted 19 Sep 2021cs.LG cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Recent advancements in self-supervised learning (SSL) made it possible to learn generalizable visual representations from unlabeled data. The performance of Deep Learning models fine-tuned on pretrained SSL representations is on par with models fine-tuned on the state-of-the-art supervised learning (SL) representations. Irrespective of the progress made in SSL, its generalizability has not been studied extensively. In this article, we perform a deeper analysis of the generalizability of pretrained SSL and SL representations by conducting a domain-based study for transfer learning classification tasks. The representations are learned from the ImageNet source data, which are then fine-tuned using two types of target datasets: similar to the source dataset, and significantly different from the source dataset. We study generalizability of the SSL and SL-based models via their prediction accuracy as well as prediction confidence. In addition to this, we analyze the attribution of the final convolutional layer of these models to understand how they reason about the semantic identity of the data. We show that the SSL representations are more generalizable as compared to the SL representations. We explain the generalizability of the SSL representations by investigating its invariance property, which is shown to be better than that observed in the SL representations.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 59 of its 92 references that are in the map. First posted on arXiv 19 Sep 2021.

Builds on

59 of 92 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2017Mask R-CNNcited by 7,259
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
38 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  13. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  14. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  15. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  16. 2014How transferable are features in deep neural networks?cited by 1,216
  17. 2016Colorful Image Colorizationcited by 1,113
  18. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  19. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  20. 2018Image Transformercited by 770
  21. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  22. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  23. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  24. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 575
  25. 2018Do Better ImageNet Models Transfer Better?cited by 560
  26. 2020FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Trackingcited by 489
  27. 2019Selective Kernel Networkscited by 401
  28. 2018Rethinking ImageNet Pre-trainingcited by 376
  29. 2020Rethinking Pre-training and Self-trainingcited by 352
  30. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  31. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  32. 2016Learning Representations for Automatic Colorizationcited by 251
  33. 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
  34. 2017Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localizationcited by 207
  35. 2017Multi-task Self-Supervised Visual Learningcited by 200
  36. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  37. 2016What makes ImageNet good for transfer learning?cited by 192
  38. 2018Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learningcited by 174

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 3; 2025: 3; 2026: 1.

18 papers.

Sort:

  1. 2023Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalizationcited by 28
  2. 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?cited by 19
  3. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  4. 2024Graph Neural Networks in Histopathology: Emerging Trends and Future Directionscited by 13
  5. 2023A Review of Predictive and Contrastive Self-supervised Learning for Medical Imagescited by 10
  6. 2023MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstractioncited by 7
  7. 2023Rethinking Self-Supervised Visual Representation Learning in Pre-training for 3D Human Pose and Shape Estimationcited by 7
  8. 2023Self-supervision for medical image classification: state-of-the-art performance with ~100 labeled training samples per classcited by 7
  9. 2024PolyCL: Contrastive Learning for Polymer Representation Learning via Explicit and Implicit Augmentationscited by 7
  10. 2023Understanding Calibration of Deep Neural Networks for Medical Image Classificationcited by 5
  11. 2022Dyna-DM: Dynamic Object-aware Self-supervised Monocular Depth Mapscited by 3
  12. 2024Self-Supervised Learning for Text Recognition: A Critical Surveycited by 3
6 more
  1. 2023Dual-Query Multiple Instance Learning for Dynamic Meta-Embedding based Tumor Classificationcited by 1
  2. 2023Efficient, Self-Supervised Human Pose Estimation with Inductive Prior Tuningcited by 1
  3. 2025Structure-preserving contrastive learning for spatial time seriescited by 1
  4. 20222nd Place Solution for ICCV 2021 VIPriors Image Classification Challenge: An Attract-and-Repulse Learning Approachcited by 0
  5. 2025Unlocking Generalization in Polyp Segmentation with DINO Self-Attention "keys"cited by 0
  6. 2026A Cautionary Tale of Self-Supervised Learning for Imaging Biomarkers: Alzheimer's Disease Case Studycited by 0