mlsanityData to 7 Oct 2026

How Well Do Self-Supervised Models Transfer?

arXiv 2011.13377 ↗First posted 26 Nov 2020cs.CVcited by 213

Not on the map. Why?

Abstract (arXiv)

Self-supervised visual representation learning has seen huge progress recently, but no large scale evaluation has compared the many models now available. We evaluate the transfer performance of 13 top self-supervised models on 40 downstream tasks, including many-shot and few-shot recognition, object detection, and dense prediction. We compare their performance to a supervised baseline and show that on most tasks the best self-supervised models outperform supervision, confirming the recently observed trend in the literature. We find ImageNet Top-1 accuracy to be highly correlated with transfer to many-shot recognition, but increasingly less so for few-shot, object detection and dense prediction. No single self-supervised method dominates overall, suggesting that universal pre-training is still unsolved. Our analysis of features suggests that top self-supervised learners fail to preserve colour information as well as supervised alternatives, but tend to induce better classifier calibration, and less attentive overfitting than supervised learners.

Key facts. Cited by 213 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 69; 2023: 41; 2024: 37; 2025: 19; 2026: 15.

213 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2023Improving CLIP Training with Language Rewritescited by 216
  3. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  4. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  5. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  6. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  7. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  8. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  9. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  10. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 123
  11. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  12. 2022Human alignment of neural network representationscited by 74
38 more
  1. 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
  2. 2021Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codescited by 71
  3. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  4. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  5. 2021High Fidelity Visualization of What Your Self-Supervised Representation Knows Aboutcited by 64
  6. 2021Do Different Tracking Tasks Require Different Appearance Models?cited by 62
  7. 2021Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Typescited by 54
  8. 2021Self-Supervised Training Enhances Online Continual Learningcited by 53
  9. 2021PASS: An ImageNet replacement for self-supervised pretraining without humanscited by 51
  10. 2021Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuningcited by 50
  11. 2021Revisiting Contrastive Methods for Unsupervised Learning of Visual Representationscited by 48
  12. 2021Self-Supervised Learning Disentangled Group Representation as Featurecited by 47
  13. 2021Neural Mean Discrepancy for Efficient Out-of-Distribution Detectioncited by 42
  14. 2022Robust and Efficient Medical Imaging with Self-Supervisioncited by 40
  15. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  16. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  17. 2023Mixed Autoencoder for Self-supervised Visual Representation Learningcited by 38
  18. 2021Contrasting Contrastive Self-Supervised Representation Learning Pipelinescited by 37
  19. 2022Understanding Cross-Domain Few-Shot Learning Based on Domain Similarity and Few-Shot Difficultycited by 37
  20. 2022Rethinking the Augmentation Module in Contrastive Learning: Learning Hierarchical Augmentation Invariance with Expanded Viewscited by 37
  21. 2021Revisiting the Transferability of Supervised Pretraining: an MLP Perspectivecited by 36
  22. 2022A Unified View of Masked Image Modelingcited by 36
  23. 2023Does progress on ImageNet transfer to real-world datasets?cited by 35
  24. 2023Exploring the Limits of Deep Image Clustering using Pretrained Modelscited by 33
  25. 2021A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysiscited by 31
  26. 2022Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Spacecited by 30
  27. 2024Review of multimodal machine learning approaches in healthcarecited by 30
  28. 2021How Well Does Self-Supervised Pre-Training Perform with Streaming Data?cited by 29
  29. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  30. 2022CorruptEncoder: Data Poisoning based Backdoor Attacks to Contrastive Learningcited by 28
  31. 2023A Systematic Review of Few-Shot Learning in Medical Imagingcited by 28
  32. 2025Do Multiple Instance Learning Models Transfer?cited by 28
  33. 2021Max-Margin Contrastive Learningcited by 27
  34. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  35. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  36. 2022Enhance the Visual Representation via Discrete Adversarial Trainingcited by 23
  37. 2022Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domaincited by 22
  38. 2023MaskCon: Masked Contrastive Learning for Coarse-Labelled Datasetcited by 21

50 most cited of 213 All 213 (JSON)