# Building a General SimCLR Self-Supervised Foundation Model Across Neurological Diseases to Advance 3D Brain MRI Diagnoses

arXiv: https://arxiv.org/abs/2509.10620 · HTML: https://mlsanity.com/paper/2509.10620/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 44 references that are in the map.
- First posted on arXiv 12 Sep 2025.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining](https://mlsanity.com/paper/2107.09559/) (2021)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [Brain Tumor Segmentation with Deep Neural Networks](https://mlsanity.com/paper/1505.03540/) (2015)
- [Exploring scalable medical image encoders beyond text supervision](https://mlsanity.com/paper/2401.10815/) (2024)
- [SynthStrip: Skull-Stripping for Any Brain Image](https://mlsanity.com/paper/2203.09974/) (2022)
- [Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker](https://mlsanity.com/paper/1612.02572/) (2016)
- [Contrastive Learning with Continuous Proxy Meta-Data for 3D MRI Classification](https://mlsanity.com/paper/2106.08808/) (2021)
- [Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification](https://mlsanity.com/paper/1701.06643/) (2017)
- [BrainSegFounder: Towards 3D Foundation Models for Neuroimage Segmentation](https://mlsanity.com/paper/2406.10395/) (2024)
- [A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Experts](https://mlsanity.com/paper/2405.10246/) (2024)

## Built on by (most cited first, 10 of 10)

- [BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning](https://mlsanity.com/paper/2604.27277/) (2026)
- [A Lightweight Medical Image Classification Framework via Self-Supervised Contrastive Learning and Quantum-Enhanced Feature Modeling](https://mlsanity.com/paper/2601.16608/) (2026)
- [GeoSAE: Geometric Prior-Guided Layer-Wise Sparse Autoencoder Annotation of Brain MRI Foundation Models](https://mlsanity.com/paper/2605.01829/) (2026)
- [VolTA-3D: Self-Supervised Learning for Brain MRI using 3D Volumetric Token Alignment](https://mlsanity.com/paper/2605.16775/) (2026)
- [Do 3D Medical Foundation Models See Through MRI Artifacts? A Controlled Study of Representation Robustness](https://mlsanity.com/paper/2608.06613/) (2026)
- [Bayesian meta-learning for modeling Alzheimer's disease progression](https://mlsanity.com/paper/2606.02228/) (2026)
- [Contrastive Joint-Embedding Prediction for Representation Learning in Structural MRI](https://mlsanity.com/paper/2607.11962/) (2026)
- [A continually expandable foundation model for brain MRI](https://mlsanity.com/paper/2608.08319/) (2026)
- [GenFAR: A generalized representation of brain structure, derived from 49,246 multi-cohort MRIs via deep learning](https://mlsanity.com/paper/2608.12185/) (2026)
- [Quantifying privacy risks in foundation model representations of brain MRI](https://mlsanity.com/paper/2610.20494/) (2026)

## Abstract (arXiv)

3D structural Magnetic Resonance Imaging (MRI) brain scans are commonly acquired in clinical settings to monitor a wide range of neurological conditions, including neurodegenerative disorders and stroke. While deep learning models have shown promising results analyzing 3D MRI across a number of brain imaging tasks, most are highly tailored for specific tasks with limited labeled data, and are not able to generalize across tasks and/or populations. The development of self-supervised learning (SSL) has enabled the creation of large medical foundation models that leverage diverse, unlabeled datasets ranging from healthy to diseased data, showing significant success in 2D medical imaging applications. However, even the very few foundation models for 3D brain MRI that have been developed remain limited in resolution, scope, or accessibility. In this work, we present a general, high-resolution SimCLR-based SSL foundation model for 3D brain structural MRI, pre-trained on 18,759 patients (44,958 scans) from 11 publicly available datasets spanning diverse neurological diseases. We compare our model to Masked Autoencoders (MAE), as well as two supervised baselines, on four diverse downstream prediction tasks in both in-distribution and out-of-distribution settings. Our fine-tuned SimCLR model outperforms all other models across all tasks. Notably, our model still achieves superior performance when fine-tuned using only 20% of labeled training samples for predicting Alzheimer's disease. We use publicly available code and data, and release our trained model at https://github.com/emilykaczmarek/3D-Neuro-SimCLR, contributing a broadly applicable and accessible foundation model for clinical brain MRI analysis.