# Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigms

arXiv: https://arxiv.org/abs/2007.04234 · HTML: https://mlsanity.com/paper/2007.04234/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2020.

## Built on by (most cited first, 23 of 23)

- [When Does Contrastive Visual Representation Learning Work?](https://mlsanity.com/paper/2105.05837/) (2021)
- [Self-Supervised Pretraining Improves Self-Supervised Pretraining](https://mlsanity.com/paper/2103.12718/) (2021)
- [Deep Model Reassembly](https://mlsanity.com/paper/2210.17409/) (2022)
- [Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets](https://mlsanity.com/paper/2310.04292/) (2023)
- [The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning](https://mlsanity.com/paper/2303.00106/) (2023)
- [How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?](https://mlsanity.com/paper/2203.14221/) (2022)
- [Augment with Care: Contrastive Learning for Combinatorial Problems](https://mlsanity.com/paper/2202.08396/) (2022)
- [How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?](https://mlsanity.com/paper/2501.11253/) (2025)
- [Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning](https://mlsanity.com/paper/2212.08378/) (2022)
- [A Survey of the Self Supervised Learning Mechanisms for Vision Transformers](https://mlsanity.com/paper/2408.17059/) (2024)
- [Self-supervised Regularization for Text Classification](https://mlsanity.com/paper/2103.05231/) (2021)
- [Shifting to Machine Supervision: Annotation-Efficient Semi and Self-Supervised Learning for Automatic Medical Image Segmentation and Classification](https://mlsanity.com/paper/2311.10319/) (2023)
- [Unsupervised Knowledge-Transfer for Learned Image Reconstruction](https://mlsanity.com/paper/2107.02572/) (2021)
- [AdapterFL: Adaptive Heterogeneous Federated Learning for Resource-constrained Mobile Computing Systems](https://mlsanity.com/paper/2311.14037/) (2023)
- [SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning](https://mlsanity.com/paper/2504.05706/) (2025)
- [Comparison of semi-supervised learning methods for High Content Screening quality control](https://mlsanity.com/paper/2208.04592/) (2022)
- [Tradeoffs Between Contrastive and Supervised Learning: An Empirical Study](https://mlsanity.com/paper/2112.05340/) (2021)
- [JOSENet: A Joint Stream Embedding Network for Violence Detection in Surveillance Videos](https://mlsanity.com/paper/2405.02961/) (2024)
- [IAdet: Simplest human-in-the-loop object detection](https://mlsanity.com/paper/2307.01582/) (2023)
- [Bridging the Gap Between Multimodal Foundation Models and World Models](https://mlsanity.com/paper/2510.03727/) (2025)
- [Self-Supervised Learning Featuring Small-Scale Image Dataset for Treatable Retinal Diseases Classification](https://mlsanity.com/paper/2404.10166/) (2024)
- [Transfer or Self-Supervised? Bridging the Performance Gap in Medical Imaging](https://mlsanity.com/paper/2407.05592/) (2024)
- [Detection of Animal Movement from Weather Radar using Self-Supervised Learning](https://mlsanity.com/paper/2408.04424/) (2024)

## Abstract (arXiv)

Pretraining has become a standard technique in computer vision and natural language processing, which usually helps to improve performance substantially. Previously, the most dominant pretraining method is transfer learning (TL), which uses labeled data to learn a good representation network. Recently, a new pretraining approach -- self-supervised learning (SSL) -- has demonstrated promising results on a wide range of applications. SSL does not require annotated labels. It is purely conducted on input data by solving auxiliary tasks defined on the input data examples. The current reported results show that in certain applications, SSL outperforms TL and the other way around in other applications. There has not been a clear understanding on what properties of data and tasks render one approach outperforms the other. Without an informed guideline, ML researchers have to try both methods to find out which one is better empirically. It is usually time-consuming to do so. In this work, we aim to address this problem. We perform a comprehensive comparative study between SSL and TL regarding which one works better under different properties of data and tasks, including domain difference between source and target tasks, the amount of pretraining data, class imbalance in source data, and usage of target data for additional pretraining, etc. The insights distilled from our comparative studies can help ML researchers decide which method to use based on the properties of their applications.