mlsanityData to 7 Oct 2026

Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigms

arXiv 2007.04234 ↗First posted 19 Jun 2020cs.CV cs.LG stat.MLcited by 23

Not on the map. Why?

Abstract (arXiv)

Pretraining has become a standard technique in computer vision and natural language processing, which usually helps to improve performance substantially. Previously, the most dominant pretraining method is transfer learning (TL), which uses labeled data to learn a good representation network. Recently, a new pretraining approach -- self-supervised learning (SSL) -- has demonstrated promising results on a wide range of applications. SSL does not require annotated labels. It is purely conducted on input data by solving auxiliary tasks defined on the input data examples. The current reported results show that in certain applications, SSL outperforms TL and the other way around in other applications. There has not been a clear understanding on what properties of data and tasks render one approach outperforms the other. Without an informed guideline, ML researchers have to try both methods to find out which one is better empirically. It is usually time-consuming to do so. In this work, we aim to address this problem. We perform a comprehensive comparative study between SSL and TL regarding which one works better under different properties of data and tasks, including domain difference between source and target tasks, the amount of pretraining data, class imbalance in source data, and usage of target data for additional pretraining, etc. The insights distilled from our comparative studies can help ML researchers decide which method to use based on the properties of their applications.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 5; 2023: 5; 2024: 5; 2025: 3.

23 papers.

Sort:

By area:

  1. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  2. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 81
  3. 2022Deep Model Reassemblycited by 78
  4. 2023Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasetscited by 37
  5. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  6. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  7. 2022Augment with Care: Contrastive Learning for Combinatorial Problemscited by 22
  8. 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?cited by 19
  9. 2022Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learningcited by 9
  10. 2024A Survey of the Self Supervised Learning Mechanisms for Vision Transformerscited by 8
  11. 2021Self-supervised Regularization for Text Classificationcited by 5
  12. 2023Shifting to Machine Supervision: Annotation-Efficient Semi and Self-Supervised Learning for Automatic Medical Image Segmentation and Classificationcited by 5
11 more
  1. 2021Unsupervised Knowledge-Transfer for Learned Image Reconstructioncited by 4
  2. 2023AdapterFL: Adaptive Heterogeneous Federated Learning for Resource-constrained Mobile Computing Systemscited by 4
  3. 2025SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learningcited by 4
  4. 2022Comparison of semi-supervised learning methods for High Content Screening quality controlcited by 3
  5. 2021Tradeoffs Between Contrastive and Supervised Learning: An Empirical Studycited by 2
  6. 2024JOSENet: A Joint Stream Embedding Network for Violence Detection in Surveillance Videoscited by 2
  7. 2023IAdet: Simplest human-in-the-loop object detectioncited by 1
  8. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  9. 2024Self-Supervised Learning Featuring Small-Scale Image Dataset for Treatable Retinal Diseases Classificationcited by 0
  10. 2024Transfer or Self-Supervised? Bridging the Performance Gap in Medical Imagingcited by 0
  11. 2024Detection of Animal Movement from Weather Radar using Self-Supervised Learningcited by 0