# Mitigating Overfitting in Medical Imaging: Self-Supervised Pretraining vs. ImageNet Transfer Learning for Dermatological Diagnosis

arXiv: https://arxiv.org/abs/2505.16773 · HTML: https://mlsanity.com/paper/2505.16773/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 10 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 5 of 5)

- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [The Road Less Scheduled](https://mlsanity.com/paper/2405.15682/) (2024)
- [Source Matters: Source Dataset Impact on Model Robustness in Medical Imaging](https://mlsanity.com/paper/2403.04484/) (2024)
- [Concordance in basal cell carcinoma diagnosis. Building a proper ground truth to train Artificial Intelligence tools](https://mlsanity.com/paper/2406.18240/) (2024)

## Abstract (arXiv)

Deep learning has transformed computer vision but relies heavily on large labeled datasets and computational resources. Transfer learning, particularly fine-tuning pretrained models, offers a practical alternative; however, models pretrained on natural image datasets such as ImageNet may fail to capture domain-specific characteristics in medical imaging. This study introduces an unsupervised learning framework that extracts high-value dermatological features instead of relying solely on ImageNet-based pretraining. We employ a Variational Autoencoder (VAE) trained from scratch on a proprietary dermatological dataset, allowing the model to learn a structured and clinically relevant latent space. This self-supervised feature extractor is then compared to an ImageNet-pretrained backbone under identical classification conditions, highlighting the trade-offs between general-purpose and domain-specific pretraining. Our results reveal distinct learning patterns. The self-supervised model achieves a final validation loss of 0.110 (-33.33%), while the ImageNet-pretrained model stagnates at 0.100 (-16.67%), indicating overfitting. Accuracy trends confirm this: the self-supervised model improves from 45% to 65% (+44.44%) with a near-zero overfitting gap, whereas the ImageNet-pretrained model reaches 87% (+50.00%) but plateaus at 75% (+19.05%), with its overfitting gap increasing to +0.060. These findings suggest that while ImageNet pretraining accelerates convergence, it also amplifies overfitting on non-clinically relevant features. In contrast, self-supervised learning achieves steady improvements, stronger generalization, and superior adaptability, underscoring the importance of domain-specific feature extraction in medical imaging.