# Evaluating Self-Supervised Learning in Medical Imaging: A Benchmark for Robustness, Generalizability, and Multi-Domain Impact

arXiv: https://arxiv.org/abs/2412.19124 · HTML: https://mlsanity.com/paper/2412.19124/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 76 references that are in the map.
- First posted on arXiv 26 Dec 2024.

## Builds on (most cited first, 30 of 47)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [SSD: A Unified Framework for Self-Supervised Outlier Detection](https://mlsanity.com/paper/2103.12051/) (2021)
- [Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?](https://mlsanity.com/paper/1706.00712/) (2017)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Scaling and Benchmarking Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1905.01235/) (2019)
- [Benchmarking Self-Supervised Learning on Diverse Pathology Datasets](https://mlsanity.com/paper/2212.04690/) (2022)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization](https://mlsanity.com/paper/2009.12829/) (2020)
- [Delving Deep into the Generalization of Vision Transformers under Distribution Shifts](https://mlsanity.com/paper/2106.07617/) (2021)
- [Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification](https://mlsanity.com/paper/2210.12843/) (2022)
- [Is it Time to Replace CNNs with Transformers for Medical Images?](https://mlsanity.com/paper/2108.09038/) (2021)
- [A scoping review of transfer learning research on medical image analysis using ImageNet](https://mlsanity.com/paper/2004.13175/) (2020)
- [Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is All You Need](https://mlsanity.com/paper/2302.02615/) (2023)
- [AIROGS: Artificial Intelligence for RObust Glaucoma Screening Challenge](https://mlsanity.com/paper/2302.01738/) (2023)
- [Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis](https://mlsanity.com/paper/2107.02568/) (2021)

## Built on by (most cited first, 2 of 2)

- [Vision Foundry: A System for Training Foundational Vision AI Models](https://mlsanity.com/paper/2512.11837/) (2025)
- [Robust Cross-Domain Generalization Using Unlabeled Target Data with Source-Domain Supervision](https://mlsanity.com/paper/2605.29122/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) has emerged as a promising paradigm in medical imaging, addressing the chronic challenge of limited labeled data in healthcare settings. While SSL has shown impressive results, existing studies in the medical domain are often limited in scope, focusing on specific datasets or modalities, or evaluating only isolated aspects of model performance. This fragmented evaluation approach poses a significant challenge, as models deployed in critical medical settings must not only achieve high accuracy but also demonstrate robust performance and generalizability across diverse datasets and varying conditions. To address this gap, we present a comprehensive evaluation of SSL methods within the medical domain, with a particular focus on robustness and generalizability. Using the MedMNIST dataset collection as a standardized benchmark, we evaluate 8 major SSL methods across 11 different medical datasets. Our study provides an in-depth analysis of model performance in both in-domain scenarios and the detection of out-of-distribution (OOD) samples, while exploring the effect of various initialization strategies, model architectures, and multi-domain pre-training. We further assess the generalizability of SSL methods through cross-dataset evaluations and the in-domain performance with varying label proportions (1%, 10%, and 100%) to simulate real-world scenarios with limited supervision. We hope this comprehensive benchmark helps practitioners and researchers make more informed decisions when applying SSL methods to medical applications.