# Interactive 3D Medical Image Segmentation with SAM 2

arXiv: https://arxiv.org/abs/2408.02635 · HTML: https://mlsanity.com/paper/2408.02635/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Builds on (most cited first, 29 of 29)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [Deep Interactive Object Selection](https://mlsanity.com/paper/1603.04042/) (2016)
- [Medical SAM 2: Segment medical images as video via Segment Anything Model 2](https://mlsanity.com/paper/2408.00874/) (2024)
- [Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications](https://mlsanity.com/paper/2304.05750/) (2023)
- [Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++](https://mlsanity.com/paper/1803.09693/) (2018)
- [DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation](https://mlsanity.com/paper/1707.00652/) (2017)
- [Annotating Object Instances with a Polygon-RNN](https://mlsanity.com/paper/1704.05548/) (2017)
- [SAM on Medical Images: A Comprehensive Study on Three Prompt Modes](https://mlsanity.com/paper/2305.00035/) (2023)
- [DeepCut: Object Segmentation from Bounding Box Annotations using Convolutional Neural Networks](https://mlsanity.com/paper/1605.07866/) (2016)
- [SAM Struggles in Concealed Scenes -- Empirical Study on Segment Anything](https://mlsanity.com/paper/2304.06022/) (2023)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model](https://mlsanity.com/paper/2304.05396/) (2023)
- [Can SAM Segment Polyps?](https://mlsanity.com/paper/2304.07583/) (2023)
- [Iteratively-Refined Interactive 3D Medical Image Segmentation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1911.10334/) (2019)
- [SAMM (Segment Any Medical Model): A 3D Slicer Integration to SAM](https://mlsanity.com/paper/2304.05622/) (2023)
- [Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomography](https://mlsanity.com/paper/2405.18356/) (2024)
- [Segment anything model 2: an application to 2D and 3D medical images](https://mlsanity.com/paper/2408.00756/) (2024)
- [Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomy](https://mlsanity.com/paper/2311.13964/) (2023)
- [Iterative Interaction Training for Segmentation Editing Networks](https://mlsanity.com/paper/1807.08555/) (2018)
- [Boundary-aware Supervoxel-level Iteratively Refined Interactive 3D Image Segmentation with Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2303.10692/) (2023)
- [Interactive Medical Image Segmentation with Self-Adaptive Confidence Calibration](https://mlsanity.com/paper/2111.07716/) (2021)

## Built on by (most cited first, 17 of 17)

- [Segment Anything in Medical Images and Videos: Benchmark and Deployment](https://mlsanity.com/paper/2408.03322/) (2024)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [Unleashing the Potential of SAM2 for Biomedical Images and Videos: A Survey](https://mlsanity.com/paper/2408.12889/) (2024)
- [Polyp SAM 2: Advancing Zero shot Polyp Segmentation in Colorectal Cancer Detection](https://mlsanity.com/paper/2408.05892/) (2024)
- [Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment](https://mlsanity.com/paper/2410.15744/) (2024)
- [CellSeg1: Robust Cell Segmentation with One Training Image](https://mlsanity.com/paper/2412.01410/) (2024)
- [A Short Review and Evaluation of SAM2's Performance in 3D CT Image Segmentation](https://mlsanity.com/paper/2408.11210/) (2024)
- [Accelerating Volumetric Medical Image Annotation via Short-Long Memory SAM 2](https://mlsanity.com/paper/2505.01854/) (2025)
- [Segment Any-Quality Images with Generative Latent Space Enhancement](https://mlsanity.com/paper/2503.12507/) (2025)
- [Semantic-Guided Diffusion Model for Single-Step Image Super-Resolution](https://mlsanity.com/paper/2505.07071/) (2025)
- [SurgiSAM2: Fine-tuning a foundational model for surgical video anatomy segmentation and detection](https://mlsanity.com/paper/2503.03942/) (2025)
- [SAM2-SGP: Enhancing SAM2 for Medical Image Segmentation via Support-Set Guided Prompting](https://mlsanity.com/paper/2506.19658/) (2025)
- [Towards user-centered interactive medical image segmentation in VR with an assistive AI agent](https://mlsanity.com/paper/2505.07214/) (2025)
- [Towards Better Ultrasound Video Segmentation Foundation Model: An Empirical study on SAM2 Finetuning from Data Perspective](https://mlsanity.com/paper/2511.05731/) (2025)
- [Towards Any-Quality Image Segmentation via Generative and Adaptive Latent Space Enhancement](https://mlsanity.com/paper/2601.02018/) (2026)
- [SAMRI-3D: Adapting SAM2 for 3D MRI Segmentation with Global Volume Tokens](https://mlsanity.com/paper/2607.18014/) (2026)
- [LeCor: Learning to Be Corrected by Meta-Learned Test-Time Training for Interactive 3D Lung-Tumour Segmentation](https://mlsanity.com/paper/2609.09477/) (2026)

## Abstract (arXiv)

Interactive medical image segmentation (IMIS) has shown significant potential in enhancing segmentation accuracy by integrating iterative feedback from medical professionals. However, the limited availability of enough 3D medical data restricts the generalization and robustness of most IMIS methods. The Segment Anything Model (SAM), though effective for 2D images, requires expensive semi-auto slice-by-slice annotations for 3D medical images. In this paper, we explore the zero-shot capabilities of SAM 2, the next-generation Meta SAM model trained on videos, for 3D medical image segmentation. By treating sequential 2D slices of 3D images as video frames, SAM 2 can fully automatically propagate annotations from a single frame to the entire 3D volume. We propose a practical pipeline for using SAM 2 in 3D medical image segmentation and present key findings highlighting its efficiency and potential for further optimization. Concretely, numerical experiments on the BraTS2020 and the medical segmentation decathlon datasets demonstrate that SAM 2 still has a gap with supervised methods but can narrow the gap in specific settings and organ types, significantly reducing the annotation burden on medical professionals. Our code will be open-sourced and available at https://github.com/Chuyun-Shen/SAM_2_Medical_3D.