mlsanityData to 14 Oct 2026

An OpenMind for 3D medical vision self-supervised learning

arXiv 2412.17041 ↗First posted 22 Dec 2024cs.CV cs.AI cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

The field of self-supervised learning (SSL) for 3D medical images lacks consistency and standardization. While many methods have been developed, it is impossible to identify the current state-of-the-art, due to i) varying and small pretraining datasets, ii) varying architectures, and iii) being evaluated on differing downstream datasets. In this paper, we bring clarity to this field and lay the foundation for further method advancements through three key contributions: We a) publish the largest publicly available pre-training dataset comprising 114k 3D brain MRI volumes, enabling all practitioners to pre-train on a large-scale dataset. We b) benchmark existing 3D self-supervised learning methods on this dataset for a state-of-the-art CNN and Transformer architecture, clarifying the state of 3D SSL pre-training. Among many findings, we show that pre-trained methods can exceed a strong from-scratch nnU-Net ResEnc-L baseline. Lastly, we c) publish the code of our pre-training and fine-tuning frameworks and provide the pre-trained models created during the benchmarking process to facilitate rapid adoption and reproduction.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 33 of its 723 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

33 of 723 references.

References · 33

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  5. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  6. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  7. 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 613
  8. 2021The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classificationcited by 518
  9. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 422
  10. 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 351
  11. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  12. 2024nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentationcited by 240
21 more
  1. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 199
  2. 2019The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challengecited by 190
  3. 2022ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation datasetcited by 106
  4. 2023STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-trainingcited by 100
  5. 2023Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modelingcited by 94
  6. 2024VoCo: A Simple-yet-Effective Volume Contrastive Learning Framework for 3D Medical Image Analysiscited by 75
  7. 2022Masked Image Modeling Advances 3D Medical Image Analysiscited by 72
  8. 2023AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weekscited by 52
  9. 2023The TopCoW Challenge -- Topology-Aware Circle of Willis Segmentation for CT and MR Angiographycited by 50
  10. 2023MultiTalent: A Multi-Dataset Approach to Medical Image Segmentationcited by 45
  11. 2018Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networkscited by 43
  12. 2023Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-trainingcited by 37
  13. 2024Revisiting MAE pre-training for 3D medical image segmentationcited by 35
  14. 2024Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?cited by 31
  15. 2019Deep Learning Enables Automatic Detection and Segmentation of Brain Metastases on Multi-Sequence MRIcited by 30
  16. 2023MIS-FM: 3D Medical Image Segmentation using Foundation Models Pretrained on a Large-Scale Unannotated Datasetcited by 26
  17. 2023Advancing Volumetric Medical Image Segmentation via Global-Local Masked Autoencodercited by 21
  18. 2024AMAES: Augmented Masked Autoencoder Pretraining on Public Brain MRI Data for 3D-Native Segmentationcited by 17
  19. 2024HySparK: Hybrid Sparse Masking for Large Scale Medical Image Pre-Trainingcited by 12
  20. 2024The Imaging Database for Epilepsy And Surgery (IDEAS)cited by 6
  21. 2024UltraCortex: Submillimeter Ultra-High Field 9.4 T Brain MR Image Collection and Manual Cortical Segmentationscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 16.

26 papers.

Sort:

By area:

  1. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  2. 2025A generalizable 3D framework and model for self-supervised learning in medical imagingcited by 31
  3. 2025Comprehensive language-image pre-training for 3D medical image understandingcited by 22
  4. 2025A large-scale heterogeneous 3D magnetic resonance brain imaging dataset for self-supervised learningcited by 8
  5. 2025Scaling Self-Supervised and Cross-Modal Pretraining for Volumetric CT Transformerscited by 7
  6. 2025MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentationcited by 5
  7. 2026Towards Brain MRI Foundation Models for the Clinic: Findings from the FOMO25 Challengecited by 3
  8. 2026Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Developmentcited by 2
  9. 2025Consistent View Alignment Improves Foundation Models for 3D Medical Image Segmentationcited by 1
  10. 2025TomoGraphView: 3D Medical Image Classification with Omnidirectional Slice Representations and Graph Neural Networkscited by 1
  11. 2026From 100,000+ images to winning the first brain MRI foundation model challenges: Sharing lessons and modelscited by 1
  12. 2026Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentationcited by 1
14 more
  1. 2026Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervisioncited by 1
  2. 2026Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Modelscited by 1
  3. 2026Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generationcited by 1
  4. 2025Learning from Anatomy: Supervised Anatomical Pretraining (SAP) for Improved Metastatic Bone Disease Segmentation in Whole-Body MRIcited by 0
  5. 2025Large-scale modality-invariant foundation models for brain MRI analysis: Application to lesion segmentationcited by 0
  6. 2026FDIF: Formula-Driven supervised Learning with Implicit Functions for 3D Medical Image Segmentationcited by 0
  7. 2026CORA: Generalizable coronary artery disease assessment and risk stratification from coronary CT angiography using pathology-centric representation learningcited by 0
  8. 2026Knowledge Transfer Scaling Laws for 3D Medical Imagingcited by 0
  9. 2026BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysiscited by 0
  10. 2026A continually expandable foundation model for brain MRIcited by 0
  11. 2026Evaluating the Generalization of Neuroimaging Foundation Models on African Brain MRIcited by 0
  12. 2026A generalizable structural brain MRI foundation model built through dual-priority federated pretrainingcited by 0
  13. 2026Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learningcited by 0
  14. 2026Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimationcited by 0