# Transfer or Self-Supervised? Bridging the Performance Gap in Medical Imaging

arXiv: https://arxiv.org/abs/2407.05592 · HTML: https://mlsanity.com/paper/2407.05592/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 63 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [What makes ImageNet good for transfer learning?](https://mlsanity.com/paper/1608.08614/) (2016)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [A Survey on Negative Transfer](https://mlsanity.com/paper/2009.00909/) (2020)
- [Deep Learning for Medical Image Processing: Overview, Challenges and Future](https://mlsanity.com/paper/1704.06825/) (2017)
- [MedMamba: Vision Mamba for Medical Image Classification](https://mlsanity.com/paper/2403.03849/) (2024)
- [Self-Supervised Learning for Videos: A Survey](https://mlsanity.com/paper/2207.00419/) (2022)
- [Secure and Robust Machine Learning for Healthcare: A Survey](https://mlsanity.com/paper/2001.08103/) (2020)
- [Self-supervised learning methods and applications in medical imaging analysis: A survey](https://mlsanity.com/paper/2109.08685/) (2021)
- [A scoping review of transfer learning research on medical image analysis using ImageNet](https://mlsanity.com/paper/2004.13175/) (2020)
- [How Transferable Are Self-supervised Features in Medical Image Classification Tasks?](https://mlsanity.com/paper/2108.10048/) (2021)
- [A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis](https://mlsanity.com/paper/2108.05930/) (2021)
- [Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigms](https://mlsanity.com/paper/2007.04234/) (2020)
- [The Three Ghosts of Medical AI: Can the Black-Box Present Deliver?](https://mlsanity.com/paper/2012.06000/) (2020)
- [Distant Domain Transfer Learning for Medical Imaging](https://mlsanity.com/paper/2012.06346/) (2020)
- [Targeted transfer learning to improve performance in small medical physics datasets](https://mlsanity.com/paper/1912.06761/) (2019)
- [LCDctCNN: Lung Cancer Diagnosis of CT scan Images Using CNN Based Model](https://mlsanity.com/paper/2304.04814/) (2023)

## Abstract (arXiv)

Recently, transfer learning and self-supervised learning have gained significant attention within the medical field due to their ability to mitigate the challenges posed by limited data availability, improve model generalisation, and reduce computational expenses. Transfer learning and self-supervised learning hold immense potential for advancing medical research. However, it is crucial to recognise that transfer learning and self-supervised learning architectures exhibit distinct advantages and limitations, manifesting variations in accuracy, training speed, and robustness. This paper compares the performance and robustness of transfer learning and self-supervised learning in the medical field. Specifically, we pre-trained two models using the same source domain datasets with different pre-training methods and evaluated them on small-sized medical datasets to identify the factors influencing their final performance. We tested data with several common issues in medical domains, such as data imbalance, data scarcity, and domain mismatch, through comparison experiments to understand their impact on specific pre-trained models. Finally, we provide recommendations to help users apply transfer learning and self-supervised learning methods in medical areas, and build more convenient and efficient deployment strategies.