# Medical SAM 2: Segment medical images as video via Segment Anything Model 2

arXiv: https://arxiv.org/abs/2408.00874 · HTML: https://mlsanity.com/paper/2408.00874/

## Key facts

- Cited by 154 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 1 Aug 2024.

## Builds on (most cited first, 30 of 31)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [Diffusion Models for Implicit Image Segmentation Ensembles](https://mlsanity.com/paper/2112.03145/) (2021)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model](https://mlsanity.com/paper/2211.00611/) (2022)
- [Data augmentation using learned transformations for one-shot medical image segmentation](https://mlsanity.com/paper/1902.09383/) (2019)
- [Self-Supervision with Superpixels: Training Few-shot Medical Image Segmentation without Annotation](https://mlsanity.com/paper/2007.09886/) (2020)
- [MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer](https://mlsanity.com/paper/2301.11798/) (2023)
- [SAM-Med2D](https://mlsanity.com/paper/2308.16184/) (2023)
- [The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT](https://mlsanity.com/paper/2307.01984/) (2023)
- [Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstraction](https://mlsanity.com/paper/2001.00208/) (2020)
- ['Squeeze & Excite' Guided Few-Shot Segmentation of Volumetric Images](https://mlsanity.com/paper/1902.01314/) (2019)
- [Boundary and Entropy-driven Adversarial Learning for Fundus Image Segmentation](https://mlsanity.com/paper/1906.11143/) (2019)
- [Boundary-aware Transformers for Skin Lesion Segmentation](https://mlsanity.com/paper/2110.03864/) (2021)
- [One-Prompt to Segment All Medical Images](https://mlsanity.com/paper/2305.10300/) (2023)
- [Modeling the Probabilistic Distribution of Unlabeled Data forOne-shot Medical Image Segmentation](https://mlsanity.com/paper/2102.02033/) (2021)
- [Focused and Collaborative Feedback Integration for Interactive Image Segmentation](https://mlsanity.com/paper/2303.11880/) (2023)
- [Automated Skin Lesion Classification Using Ensemble of Deep Neural Networks in ISIC 2018: Skin Lesion Analysis Towards Melanoma Detection Challenge](https://mlsanity.com/paper/1901.10802/) (2019)
- [HyperSegNAS: Bridging One-Shot Neural Architecture Search with 3D Medical Image Segmentation using HyperNet](https://mlsanity.com/paper/2112.10652/) (2021)

## Built on by (most cited first, 30 of 154)

- [MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow](https://mlsanity.com/paper/2503.18968/) (2025)
- [Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning](https://mlsanity.com/paper/2408.07931/) (2024)
- [Segment Anything in Medical Images and Videos: Benchmark and Deployment](https://mlsanity.com/paper/2408.03322/) (2024)
- [MedSAM3: Delving into Segment Anything with Medical Concepts](https://mlsanity.com/paper/2511.19046/) (2025)
- [Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation](https://mlsanity.com/paper/2601.10880/) (2026)
- [vesselFM: A Foundation Model for Universal 3D Blood Vessel Segmentation](https://mlsanity.com/paper/2411.17386/) (2024)
- [Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2412.02621/) (2024)
- [Interactive 3D Medical Image Segmentation with SAM 2](https://mlsanity.com/paper/2408.02635/) (2024)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [ROS-SAM: High-Quality Interactive Segmentation for Remote Sensing Moving Object](https://mlsanity.com/paper/2503.12006/) (2025)
- [Unleashing the Potential of SAM2 for Biomedical Images and Videos: A Survey](https://mlsanity.com/paper/2408.12889/) (2024)
- [Towards a vision foundation model for comprehensive assessment of Cardiac MRI](https://mlsanity.com/paper/2410.01665/) (2024)
- [Improving SAM for Camouflaged Object Detection via Dual Stream Adapters](https://mlsanity.com/paper/2503.06042/) (2025)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for Multi-Organ Segmentation](https://mlsanity.com/paper/2506.23121/) (2025)
- [Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2](https://mlsanity.com/paper/2510.24195/) (2025)
- [UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretation](https://mlsanity.com/paper/2504.21336/) (2025)
- [Adapting Vision Foundation Models for Real-time Ultrasound Image Segmentation](https://mlsanity.com/paper/2503.24368/) (2025)
- [Polyp SAM 2: Advancing Zero shot Polyp Segmentation in Colorectal Cancer Detection](https://mlsanity.com/paper/2408.05892/) (2024)
- [PRS-Med: Position Reasoning Segmentation in Medical Imaging](https://mlsanity.com/paper/2505.11872/) (2025)
- [SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost](https://mlsanity.com/paper/2506.01304/) (2025)
- [SAMed-2: Selective Memory Enhanced Medical Segment Anything Model](https://mlsanity.com/paper/2507.03698/) (2025)
- [Memory-Augmented SAM2 for Training-Free Surgical Video Segmentation](https://mlsanity.com/paper/2507.09577/) (2025)
- [Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning](https://mlsanity.com/paper/2509.19090/) (2025)
- [RevSAM2: Prompt SAM2 for Medical Image Segmentation via Reverse-Propagation without Fine-tuning](https://mlsanity.com/paper/2409.04298/) (2024)
- [Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment](https://mlsanity.com/paper/2410.15744/) (2024)
- [MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis](https://mlsanity.com/paper/2506.08900/) (2025)
- [MedicoSAM: Robust Improvement of SAM for Medical Imaging](https://mlsanity.com/paper/2501.11734/) (2025)
- [A Short Review and Evaluation of SAM2's Performance in 3D CT Image Segmentation](https://mlsanity.com/paper/2408.11210/) (2024)
- [Evaluation Study on SAM 2 for Class-agnostic Instance-level Segmentation](https://mlsanity.com/paper/2409.02567/) (2024)

## Abstract (arXiv)

Medical image segmentation plays a pivotal role in clinical diagnostics and treatment planning, yet existing models often face challenges in generalization and in handling both 2D and 3D data uniformly. In this paper, we introduce Medical SAM 2 (MedSAM-2), a generalized auto-tracking model for universal 2D and 3D medical image segmentation. The core concept is to leverage the Segment Anything Model 2 (SAM2) pipeline to treat all 2D and 3D medical segmentation tasks as a video object tracking problem. To put it into practice, we propose a novel self-sorting memory bank mechanism that dynamically selects informative embeddings based on confidence and dissimilarity, regardless of temporal order. This mechanism not only significantly improves performance in 3D medical image segmentation but also unlocks a One-Prompt Segmentation capability for 2D images, allowing segmentation across multiple images from a single prompt without temporal relationships. We evaluated MedSAM-2 on five 2D tasks and nine 3D tasks, including white blood cells, optic cups, retinal vessels, mandibles, coronary arteries, kidney tumors, liver tumors, breast cancer, nasopharynx cancer, vestibular schwannoma, mediastinal lymph nodules, cerebral artery, inferior alveolar nerve, and abdominal organs, comparing it against state-of-the-art (SOTA) models in task-tailored, general and interactive segmentation settings. Our findings demonstrate that MedSAM-2 surpasses a wide range of existing models and updates new SOTA on several benchmarks. The code is released on the project page: https://supermedintel.github.io/Medical-SAM2/.