# SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images

arXiv: https://arxiv.org/abs/2310.15161 · HTML: https://mlsanity.com/paper/2310.15161/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [AIM: Adapting Image Models for Efficient Video Action Recognition](https://mlsanity.com/paper/2302.03024/) (2023)
- [SAM-Med2D](https://mlsanity.com/paper/2308.16184/) (2023)
- [The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT](https://mlsanity.com/paper/2307.01984/) (2023)
- [STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training](https://mlsanity.com/paper/2304.06716/) (2023)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [SAM on Medical Images: A Comprehensive Study on Three Prompt Modes](https://mlsanity.com/paper/2305.00035/) (2023)
- [MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentation](https://mlsanity.com/paper/2309.08842/) (2023)
- [3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation](https://mlsanity.com/paper/2306.13465/) (2023)
- [An automatic multi-tissue human fetal brain segmentation benchmark using the Fetal Tissue Annotation Dataset](https://mlsanity.com/paper/2010.15526/) (2020)
- [Can SAM Segment Polyps?](https://mlsanity.com/paper/2304.07583/) (2023)
- [When SAM Meets Medical Images: An Investigation of Segment Anything Model (SAM) on Multi-phase Liver Tumor Segmentation](https://mlsanity.com/paper/2304.08506/) (2023)
- [MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning](https://mlsanity.com/paper/2104.12166/) (2021)
- [SAM3D: Segment Anything Model in Volumetric Medical Images](https://mlsanity.com/paper/2309.03493/) (2023)
- [mathrmSAMMed: A medical image annotation framework based on large vision model](https://mlsanity.com/paper/2307.05617/) (2023)

## Built on by (most cited first, 30 of 107)

- [Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions](https://mlsanity.com/paper/2401.03495/) (2024)
- [MedSAM2: Segment Anything in 3D Medical Images and Videos](https://mlsanity.com/paper/2504.03600/) (2025)
- [Large-Vocabulary Segmentation for Medical Images with Text Prompts](https://mlsanity.com/paper/2312.17183/) (2023)
- [A generalizable 3D framework and model for self-supervised learning in medical imaging](https://mlsanity.com/paper/2501.11755/) (2025)
- [vesselFM: A Foundation Model for Universal 3D Blood Vessel Segmentation](https://mlsanity.com/paper/2411.17386/) (2024)
- [Show and Segment: Universal Medical Image Segmentation via In-Context Learning](https://mlsanity.com/paper/2503.19359/) (2025)
- [PRISM: A Promptable and Robust Interactive Segmentation Model with Visual Prompts](https://mlsanity.com/paper/2404.15028/) (2024)
- [FastSAM3D: An Efficient Segment Anything Model for 3D Volumetric Medical Images](https://mlsanity.com/paper/2403.09827/) (2024)
- [A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Experts](https://mlsanity.com/paper/2405.10246/) (2024)
- [SAM-Med3D-MoE: Towards a Non-Forgetting Segment Anything Model via Mixture of Experts for 3D Medical Image Segmentation](https://mlsanity.com/paper/2407.04938/) (2024)
- [Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2503.04639/) (2025)
- [CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for Multi-Organ Segmentation](https://mlsanity.com/paper/2506.23121/) (2025)
- [SegAnyPET: Universal Promptable Segmentation from Positron Emission Tomography Images](https://mlsanity.com/paper/2502.14351/) (2025)
- [Recent Advances in Medical Imaging Segmentation: A Survey](https://mlsanity.com/paper/2505.09274/) (2025)
- [WSI-SAM: Multi-resolution Segment Anything Model (SAM) for histopathology whole-slide images](https://mlsanity.com/paper/2403.09257/) (2024)
- [Brain Foundation Models with Hypergraph Dynamic Adapter for Brain Disease Analysis](https://mlsanity.com/paper/2505.00627/) (2025)
- [One registration is worth two segmentations](https://mlsanity.com/paper/2405.10879/) (2024)
- [Decipher-MR: A Vision-Language Foundation Model for 3D MRI Representations](https://mlsanity.com/paper/2509.21249/) (2025)
- [TV-SAM: Increasing Zero-Shot Segmentation Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation](https://mlsanity.com/paper/2402.15759/) (2024)
- [ONCOPILOT: A Promptable CT Foundation Model For Solid Tumor Evaluation](https://mlsanity.com/paper/2410.07908/) (2024)
- [Few-Shot Adaptation of Training-Free Foundation Model for 3D Medical Image Segmentation](https://mlsanity.com/paper/2501.09138/) (2025)
- [Conformal Lesion Segmentation for 3D Medical Images](https://mlsanity.com/paper/2510.17897/) (2025)
- [MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentation](https://mlsanity.com/paper/2512.17774/) (2025)
- [Mask-Enhanced Segment Anything Model for Tumor Lesion Semantic Segmentation](https://mlsanity.com/paper/2403.05912/) (2024)
- [Leveraging Segment Anything Model for Source-Free Domain Adaptation via Dual Feature Guided Auto-Prompting](https://mlsanity.com/paper/2505.08527/) (2025)
- [Towards Cross-Domain Single Blood Cell Image Classification via Large-Scale LoRA-based Segment Anything Model](https://mlsanity.com/paper/2408.06716/) (2024)
- [Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration](https://mlsanity.com/paper/2408.07341/) (2024)
- [ShapeMamba-EM: Fine-Tuning Foundation Model with Local Shape Descriptors and Mamba Blocks for 3D EM Image Segmentation](https://mlsanity.com/paper/2408.14114/) (2024)
- [Zero-shot Domain Generalization of Foundational Models for 3D Medical Image Segmentation: An Experimental Study](https://mlsanity.com/paper/2503.22862/) (2025)
- [MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation](https://mlsanity.com/paper/2508.11032/) (2025)

## Abstract (arXiv)

Existing volumetric medical image segmentation models are typically task-specific, excelling at specific target but struggling to generalize across anatomical structures or modalities. This limitation restricts their broader clinical use. In this paper, we introduce SAM-Med3D for general-purpose segmentation on volumetric medical images. Given only a few 3D prompt points, SAM-Med3D can accurately segment diverse anatomical structures and lesions across various modalities. To achieve this, we gather and process a large-scale 3D medical image dataset, SA-Med3D-140K, from a blend of public sources and licensed private datasets. This dataset includes 22K 3D images and 143K corresponding 3D masks. Then SAM-Med3D, a promptable segmentation model characterized by the fully learnable 3D structure, is trained on this dataset using a two-stage procedure and exhibits impressive performance on both seen and unseen segmentation targets. We comprehensively evaluate SAM-Med3D on 16 datasets covering diverse medical scenarios, including different anatomical structures, modalities, targets, and zero-shot transferability to new/unseen tasks. The evaluation shows the efficiency and efficacy of SAM-Med3D, as well as its promising application to diverse downstream tasks as a pre-trained model. Our approach demonstrates that substantial medical resources can be utilized to develop a general-purpose medical AI for various potential applications. Our dataset, code, and models are available at https://github.com/uni-medical/SAM-Med3D.