# A generalizable 3D framework and model for self-supervised learning in medical imaging

arXiv: https://arxiv.org/abs/2501.11755 · HTML: https://mlsanity.com/paper/2501.11755/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 68 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [3D Self-Supervised Methods for Medical Imaging](https://mlsanity.com/paper/2006.03829/) (2020)
- [Spreading vectors for similarity search](https://mlsanity.com/paper/1806.03198/) (2018)
- [SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images](https://mlsanity.com/paper/2310.15161/) (2023)
- [Masked Image Modeling Advances 3D Medical Image Analysis](https://mlsanity.com/paper/2204.11716/) (2022)
- [An OpenMind for 3D medical vision self-supervised learning](https://mlsanity.com/paper/2412.17041/) (2024)
- [On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learning](https://mlsanity.com/paper/2207.07875/) (2022)
- [ROOD-MRI: Benchmarking the robustness of deep learning segmentation models to out-of-distribution and corrupted data in MRI](https://mlsanity.com/paper/2203.06060/) (2022)

## Built on by (most cited first, 30 of 31)

- [Comprehensive language-image pre-training for 3D medical image understanding](https://mlsanity.com/paper/2510.15042/) (2025)
- [Revisiting 2D Foundation Models for Scalable 3D Medical Image Classification](https://mlsanity.com/paper/2512.12887/) (2025)
- [The MAMA-MIA Challenge: Advancing Generalizability and Fairness in Breast MRI Tumor Segmentation and Treatment Response Prediction](https://mlsanity.com/paper/2603.01250/) (2026)
- [Multi-encoder nnU-Net outperforms transformer models with self-supervised pretraining](https://mlsanity.com/paper/2504.03474/) (2025)
- [Live(r) Die: Predicting Survival in Colorectal Liver Metastasis](https://mlsanity.com/paper/2509.08935/) (2025)
- [Opportunistic Promptable Segmentation: Leveraging Routine Radiological Annotations to Guide 3D CT Lesion Segmentation](https://mlsanity.com/paper/2602.00309/) (2026)
- [Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision](https://mlsanity.com/paper/2603.13660/) (2026)
- [M3Net: A Macro-to-Meso-to-Micro Clinical-inspired Hierarchical 3D Network for Pulmonary Nodule Classification](https://mlsanity.com/paper/2605.12570/) (2026)
- [Vision Foundation Models in Radiology: A Scoping Review of Data, Methodology, Evaluation and Clinical Translation](https://mlsanity.com/paper/2607.07219/) (2026)
- [Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography](https://mlsanity.com/paper/2607.26196/) (2026)
- [Anatomy Contextualized Adaptation of CT Foundation Models](https://mlsanity.com/paper/2607.27154/) (2026)
- [Do Medical Foundation Models Generalize on the African Brain?](https://mlsanity.com/paper/2607.28771/) (2026)
- [Do 3D Medical Foundation Models See Through MRI Artifacts? A Controlled Study of Representation Robustness](https://mlsanity.com/paper/2608.06613/) (2026)
- [Multi-Scale Feature Fusion with Image-Driven Spatial Integration for Left Atrium Segmentation from Cardiac MRI Images](https://mlsanity.com/paper/2502.06615/) (2025)
- [Evaluating Vision Language Models (VLMs) for Radiology: A Comprehensive Analysis](https://mlsanity.com/paper/2504.16047/) (2025)
- [PRISM: A Framework Harnessing Unsupervised Visual Representations and Textual Prompts for Explainable MACE Survival Prediction from Cardiac Cine MRI](https://mlsanity.com/paper/2508.19325/) (2025)
- [Shaken or Stirred? An Analysis of MetaFormer's Token Mixing for Medical Imaging](https://mlsanity.com/paper/2510.05971/) (2025)
- [An Automated Radiomics Framework for Postoperative Survival Prediction in Colorectal Liver Metastases using Preoperative MRI](https://mlsanity.com/paper/2603.10216/) (2026)
- [FDIF: Formula-Driven supervised Learning with Implicit Functions for 3D Medical Image Segmentation](https://mlsanity.com/paper/2603.23199/) (2026)
- [3D MRI Image Pretraining via Controllable 2D Slice Navigation Task](https://mlsanity.com/paper/2605.06487/) (2026)
- [Knowledge Transfer Scaling Laws for 3D Medical Imaging](https://mlsanity.com/paper/2605.06859/) (2026)
- [MultiMedVision: Multi-Modal Medical Vision Framework](https://mlsanity.com/paper/2605.09151/) (2026)
- [Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines](https://mlsanity.com/paper/2605.23995/) (2026)
- [PatTree: a novel approach for automated creation of multimodal, graph-based patient representations for medical classification tasks](https://mlsanity.com/paper/2608.02692/) (2026)
- [Representation learning of human cortical folding to reveal long lasting neurodevelopmental signatures](https://mlsanity.com/paper/2609.05438/) (2026)
- [AWR-Net: Decoupling Anatomy and Appearance for 3D Fetal Brain Ultrasound Synthesis](https://mlsanity.com/paper/2609.22635/) (2026)
- [FLASHSWIN: Unlocking Large Windows and Dense Tokens in Swin Vision Transformers with Memory Efficient Attention](https://mlsanity.com/paper/2610.04664/) (2026)
- [Dataset-Specific Encoding of Image Content Underlies Cross-Dataset Failure in Brain MRI Foundation Models](https://mlsanity.com/paper/2610.13027/) (2026)
- [A4-MR: Abnormality-aware Anatomical Alignment for MR](https://mlsanity.com/paper/2610.13056/) (2026)
- [Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learning](https://mlsanity.com/paper/2610.13915/) (2026)

## Abstract (arXiv)

Current self-supervised learning methods for 3D medical imaging rely on simple pretext formulations and organ- or modality-specific datasets, limiting their generalizability and scalability. We present 3DINO, a cutting-edge SSL method adapted to 3D datasets, and use it to pretrain 3DINO-ViT: a general-purpose medical imaging model, on an exceptionally large, multimodal, and multi-organ dataset of ~100,000 3D medical imaging scans from over 10 organs. We validate 3DINO-ViT using extensive experiments on numerous medical imaging segmentation and classification tasks. Our results demonstrate that 3DINO-ViT generalizes across modalities and organs, including out-of-distribution tasks and datasets, outperforming state-of-the-art methods on the majority of evaluation metrics and labeled dataset sizes. Our 3DINO framework and 3DINO-ViT will be made available to enable research on 3D foundation models or further finetuning for a wide range of medical imaging applications.