# An OpenMind for 3D medical vision self-supervised learning

arXiv: https://arxiv.org/abs/2412.17041 · HTML: https://mlsanity.com/paper/2412.17041/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 723 references that are in the map.
- First posted on arXiv 22 Dec 2024.

## Builds on (most cited first, 30 of 33)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](https://mlsanity.com/paper/2404.09556/) (2024)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge](https://mlsanity.com/paper/1912.01054/) (2019)
- [ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset](https://mlsanity.com/paper/2206.06694/) (2022)
- [STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training](https://mlsanity.com/paper/2304.06716/) (2023)
- [Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling](https://mlsanity.com/paper/2301.03580/) (2023)
- [VoCo: A Simple-yet-Effective Volume Contrastive Learning Framework for 3D Medical Image Analysis](https://mlsanity.com/paper/2402.17300/) (2024)
- [Masked Image Modeling Advances 3D Medical Image Analysis](https://mlsanity.com/paper/2204.11716/) (2022)
- [AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weeks](https://mlsanity.com/paper/2305.09666/) (2023)
- [The TopCoW Challenge -- Topology-Aware Circle of Willis Segmentation for CT and MR Angiography](https://mlsanity.com/paper/2312.17670/) (2023)
- [MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation](https://mlsanity.com/paper/2303.14444/) (2023)
- [Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks](https://mlsanity.com/paper/1810.03856/) (2018)
- [Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-training](https://mlsanity.com/paper/2303.00874/) (2023)
- [Revisiting MAE pre-training for 3D medical image segmentation](https://mlsanity.com/paper/2410.23132/) (2024)
- [Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?](https://mlsanity.com/paper/2411.03670/) (2024)
- [Deep Learning Enables Automatic Detection and Segmentation of Brain Metastases on Multi-Sequence MRI](https://mlsanity.com/paper/1903.07988/) (2019)
- [MIS-FM: 3D Medical Image Segmentation using Foundation Models Pretrained on a Large-Scale Unannotated Dataset](https://mlsanity.com/paper/2306.16925/) (2023)
- [Advancing Volumetric Medical Image Segmentation via Global-Local Masked Autoencoder](https://mlsanity.com/paper/2306.08913/) (2023)
- [AMAES: Augmented Masked Autoencoder Pretraining on Public Brain MRI Data for 3D-Native Segmentation](https://mlsanity.com/paper/2408.00640/) (2024)

## Built on by (most cited first, 26 of 26)

- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [A generalizable 3D framework and model for self-supervised learning in medical imaging](https://mlsanity.com/paper/2501.11755/) (2025)
- [Comprehensive language-image pre-training for 3D medical image understanding](https://mlsanity.com/paper/2510.15042/) (2025)
- [A large-scale heterogeneous 3D magnetic resonance brain imaging dataset for self-supervised learning](https://mlsanity.com/paper/2506.14432/) (2025)
- [Scaling Self-Supervised and Cross-Modal Pretraining for Volumetric CT Transformers](https://mlsanity.com/paper/2511.17209/) (2025)
- [MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentation](https://mlsanity.com/paper/2512.17774/) (2025)
- [Towards Brain MRI Foundation Models for the Clinic: Findings from the FOMO25 Challenge](https://mlsanity.com/paper/2604.11679/) (2026)
- [Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development](https://mlsanity.com/paper/2603.27460/) (2026)
- [Consistent View Alignment Improves Foundation Models for 3D Medical Image Segmentation](https://mlsanity.com/paper/2509.13846/) (2025)
- [TomoGraphView: 3D Medical Image Classification with Omnidirectional Slice Representations and Graph Neural Networks](https://mlsanity.com/paper/2511.09605/) (2025)
- [From 100,000+ images to winning the first brain MRI foundation model challenges: Sharing lessons and models](https://mlsanity.com/paper/2601.13166/) (2026)
- [Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation](https://mlsanity.com/paper/2602.23916/) (2026)
- [Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision](https://mlsanity.com/paper/2603.13660/) (2026)
- [Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models](https://mlsanity.com/paper/2607.04199/) (2026)
- [Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation](https://mlsanity.com/paper/2608.08713/) (2026)
- [Learning from Anatomy: Supervised Anatomical Pretraining (SAP) for Improved Metastatic Bone Disease Segmentation in Whole-Body MRI](https://mlsanity.com/paper/2506.19590/) (2025)
- [Large-scale modality-invariant foundation models for brain MRI analysis: Application to lesion segmentation](https://mlsanity.com/paper/2511.11311/) (2025)
- [FDIF: Formula-Driven supervised Learning with Implicit Functions for 3D Medical Image Segmentation](https://mlsanity.com/paper/2603.23199/) (2026)
- [CORA: Generalizable coronary artery disease assessment and risk stratification from coronary CT angiography using pathology-centric representation learning](https://mlsanity.com/paper/2603.24847/) (2026)
- [Knowledge Transfer Scaling Laws for 3D Medical Imaging](https://mlsanity.com/paper/2605.06859/) (2026)
- [BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis](https://mlsanity.com/paper/2607.17782/) (2026)
- [A continually expandable foundation model for brain MRI](https://mlsanity.com/paper/2608.08319/) (2026)
- [Evaluating the Generalization of Neuroimaging Foundation Models on African Brain MRI](https://mlsanity.com/paper/2609.23983/) (2026)
- [A generalizable structural brain MRI foundation model built through dual-priority federated pretraining](https://mlsanity.com/paper/2609.27611/) (2026)
- [Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learning](https://mlsanity.com/paper/2610.13915/) (2026)
- [Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation](https://mlsanity.com/paper/2610.17573/) (2026)

## Abstract (arXiv)

The field of self-supervised learning (SSL) for 3D medical images lacks consistency and standardization. While many methods have been developed, it is impossible to identify the current state-of-the-art, due to i) varying and small pretraining datasets, ii) varying architectures, and iii) being evaluated on differing downstream datasets. In this paper, we bring clarity to this field and lay the foundation for further method advancements through three key contributions: We a) publish the largest publicly available pre-training dataset comprising 114k 3D brain MRI volumes, enabling all practitioners to pre-train on a large-scale dataset. We b) benchmark existing 3D self-supervised learning methods on this dataset for a state-of-the-art CNN and Transformer architecture, clarifying the state of 3D SSL pre-training. Among many findings, we show that pre-trained methods can exceed a strong from-scratch nnU-Net ResEnc-L baseline. Lastly, we c) publish the code of our pre-training and fine-tuning frameworks and provide the pre-trained models created during the benchmarking process to facilitate rapid adoption and reproduction.