# Segment anything model 2: an application to 2D and 3D medical images

arXiv: https://arxiv.org/abs/2408.00756 · HTML: https://mlsanity.com/paper/2408.00756/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 59 references that are in the map.
- First posted on arXiv 1 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography](https://mlsanity.com/paper/1908.06948/) (2019)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2211.15583/) (2022)
- [Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation](https://mlsanity.com/paper/2104.04691/) (2021)
- [SAM-Med2D](https://mlsanity.com/paper/2308.16184/) (2023)
- [Polyp-SAM: Transfer SAM for Polyp Segmentation](https://mlsanity.com/paper/2305.00293/) (2023)
- [SAM3D: Segment Anything Model in Volumetric Medical Images](https://mlsanity.com/paper/2309.03493/) (2023)
- [How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model](https://mlsanity.com/paper/2404.09957/) (2024)
- [Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation](https://mlsanity.com/paper/2401.13220/) (2024)
- [MMOTU: A Multi-Modality Ovarian Tumor Ultrasound Image Dataset for Unsupervised Cross-Domain Semantic Segmentation](https://mlsanity.com/paper/2207.06799/) (2022)
- [SegmentAnyBone: A Universal Model that Segments Any Bone at Any Location on MRI](https://mlsanity.com/paper/2401.12974/) (2024)
- [QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge](https://mlsanity.com/paper/2405.18435/) (2024)
- [Cataract-1K: Cataract Surgery Dataset for Scene Segmentation, Phase Recognition, and Irregularity Detection](https://mlsanity.com/paper/2312.06295/) (2023)

## Built on by (most cited first, 28 of 28)

- [Segment Anything in Medical Images and Videos: Benchmark and Deployment](https://mlsanity.com/paper/2408.03322/) (2024)
- [Serp-Mamba: Advancing High-Resolution Retinal Vessel Segmentation with Selective State-Space Model](https://mlsanity.com/paper/2409.04356/) (2024)
- [Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2412.02621/) (2024)
- [Interactive 3D Medical Image Segmentation with SAM 2](https://mlsanity.com/paper/2408.02635/) (2024)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [Unleashing the Potential of SAM2 for Biomedical Images and Videos: A Survey](https://mlsanity.com/paper/2408.12889/) (2024)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [Polyp SAM 2: Advancing Zero shot Polyp Segmentation in Colorectal Cancer Detection](https://mlsanity.com/paper/2408.05892/) (2024)
- [SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost](https://mlsanity.com/paper/2506.01304/) (2025)
- [MedicoSAM: Robust Improvement of SAM for Medical Imaging](https://mlsanity.com/paper/2501.11734/) (2025)
- [Are Vision Foundation Models Ready for Out-of-the-Box Medical Image Registration?](https://mlsanity.com/paper/2507.11569/) (2025)
- [A Short Review and Evaluation of SAM2's Performance in 3D CT Image Segmentation](https://mlsanity.com/paper/2408.11210/) (2024)
- [Automated Muscle and Fat Segmentation in Computed Tomography for Comprehensive Body Composition Analysis](https://mlsanity.com/paper/2502.09779/) (2025)
- [Accelerating Volumetric Medical Image Annotation via Short-Long Memory SAM 2](https://mlsanity.com/paper/2505.01854/) (2025)
- [SAM & SAM 2 in 3D Slicer: SegmentWithSAM Extension for Annotating Medical Images](https://mlsanity.com/paper/2408.15224/) (2024)
- [Assessing Foundational Medical 'Segment Anything' (Med-SAM1, Med-SAM2) Deep Learning Models for Left Atrial Segmentation in 3D LGE MRI](https://mlsanity.com/paper/2411.05963/) (2024)
- [SurgiSAM2: Fine-tuning a foundational model for surgical video anatomy segmentation and detection](https://mlsanity.com/paper/2503.03942/) (2025)
- [SAM2-3dMed: Empowering SAM2 for 3D Medical Image Segmentation](https://mlsanity.com/paper/2510.08967/) (2025)
- [Comparing SAM 2 and SAM 3 for Zero-Shot Segmentation of 3D Medical Data](https://mlsanity.com/paper/2511.21926/) (2025)
- [RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video](https://mlsanity.com/paper/2511.22950/) (2025)
- [Model Agnostic Preference Optimization for Medical Image Segmentation](https://mlsanity.com/paper/2512.15009/) (2025)
- [UltraSAM3: A Concept-Driven Foundation Model for Universal Ultrasound Image Segmentation](https://mlsanity.com/paper/2607.29200/) (2026)
- [Zero-shot 3D Segmentation of Abdominal Organs in CT Scans Using Segment Anything Model 2: Adapting Video Tracking Capabilities for 3D Medical Imaging](https://mlsanity.com/paper/2408.06170/) (2024)
- [Towards Better Ultrasound Video Segmentation Foundation Model: An Empirical study on SAM2 Finetuning from Data Perspective](https://mlsanity.com/paper/2511.05731/) (2025)
- [Prompting with the human-touch: evaluating model-sensitivity of foundation models for musculoskeletal CT segmentation](https://mlsanity.com/paper/2603.10541/) (2026)
- [Enhancing MedSAM with a Lightweight Box Predictor for Medical Image Segmentation](https://mlsanity.com/paper/2606.04705/) (2026)
- [Automatic bioinformatic software named entity recognition from literature](https://mlsanity.com/paper/2608.19201/) (2026)
- [Beyond the Encoder: Object-Centric Representations Emerging from Segmentation Foundation Models](https://mlsanity.com/paper/2610.18543/) (2026)

## Abstract (arXiv)

Segment Anything Model (SAM) has gained significant attention because of its ability to segment various objects in images given a prompt. The recently developed SAM 2 has extended this ability to video inputs. This opens an opportunity to apply SAM to 3D images, one of the fundamental tasks in the medical imaging field. In this paper, we extensively evaluate SAM 2's ability to segment both 2D and 3D medical images by first collecting 21 medical imaging datasets, including surgical videos, common 3D modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) as well as 2D modalities such as X-ray and ultrasound. Two evaluation settings of SAM 2 are considered: (1) multi-frame 3D segmentation, where prompts are provided to one or multiple slice(s) selected from the volume, and (2) single-frame 2D segmentation, where prompts are provided to each slice. The former only applies to videos and 3D modalities, while the latter applies to all datasets. Our results show that SAM 2 exhibits similar performance as SAM under single-frame 2D segmentation, and has variable performance under multi-frame 3D segmentation depending on the choices of slices to annotate, the direction of the propagation, the predictions utilized during the propagation, etc. We believe our work enhances the understanding of SAM 2's behavior in the medical field and provides directions for future work in adapting SAM 2 to this domain. Our code is available at: https://github.com/mazurowski-lab/segment-anything2-medical-evaluation.