# MedSAM3: Delving into Segment Anything with Medical Concepts

arXiv: https://arxiv.org/abs/2511.19046 · HTML: https://mlsanity.com/paper/2511.19046/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 62 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Builds on (most cited first, 30 of 45)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation](https://mlsanity.com/paper/2004.08790/) (2020)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers](https://mlsanity.com/paper/2108.06932/) (2021)
- [Osprey: Pixel Understanding with Visual Instruction Tuning](https://mlsanity.com/paper/2312.10032/) (2023)
- [Medical SAM 2: Segment medical images as video via Segment Anything Model 2](https://mlsanity.com/paper/2408.00874/) (2024)
- [LViT: Language meets Vision Transformer in Medical Image Segmentation](https://mlsanity.com/paper/2206.14718/) (2022)
- [BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once](https://mlsanity.com/paper/2405.12971/) (2024)

## Built on by (most cited first, 26 of 26)

- [MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution](https://mlsanity.com/paper/2604.26283/) (2026)
- [GLeVE: Graph-Guided Lesion Grounding with Proposal Verification in 3D CT](https://mlsanity.com/paper/2605.22619/) (2026)
- [Open-Ended CT Volume Segmentation with Weak Supervision from Language](https://mlsanity.com/paper/2607.25860/) (2026)
- [UltraSAM3: A Concept-Driven Foundation Model for Universal Ultrasound Image Segmentation](https://mlsanity.com/paper/2607.29200/) (2026)
- [Vision-Language Controlled Deep Unfolding for Joint Medical Image Restoration and Segmentation](https://mlsanity.com/paper/2601.23103/) (2026)
- [Promptable segmentation with region exploration enables minimal-effort expert-level prostate cancer delineation](https://mlsanity.com/paper/2602.17813/) (2026)
- [Uncertainty-Aware Concept and Motion Segmentation for Semi-Supervised Angiography Videos](https://mlsanity.com/paper/2603.00881/) (2026)
- [TubeMLLM: A Foundation Model for Topology Knowledge Exploration in Vessel-like Anatomy](https://mlsanity.com/paper/2603.09217/) (2026)
- [Hierarchical Text-Guided Brain Tumor Segmentation via Sub-Region-Aware Prompts](https://mlsanity.com/paper/2603.21083/) (2026)
- [DGRNet: Disagreement-Guided Refinement for Uncertainty-Aware Brain Tumor Segmentation](https://mlsanity.com/paper/2603.21086/) (2026)
- [Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models](https://mlsanity.com/paper/2605.20158/) (2026)
- [EchoPilot: Training-Free Ultrasound Video Segmentation via Scale-Space Semantic Prompting and Reliability-Gated Memory](https://mlsanity.com/paper/2605.25944/) (2026)
- [MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models](https://mlsanity.com/paper/2606.06760/) (2026)
- [Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs](https://mlsanity.com/paper/2606.12590/) (2026)
- [DINO-Med3D: Bridging Dimension and Domain Gaps in Volumetric Segmentation via Progressive Adaptation](https://mlsanity.com/paper/2606.18886/) (2026)
- [Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation](https://mlsanity.com/paper/2606.22963/) (2026)
- [PGE-SAM: Prompt-Guided Feature Enhancement for Interactive Segmentation under Degradation](https://mlsanity.com/paper/2606.30477/) (2026)
- [SABER: A Semantic-Aligned Brain Network Analysis Framework via Multi-scale Hypergraphs](https://mlsanity.com/paper/2607.01901/) (2026)
- [Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentation](https://mlsanity.com/paper/2607.17341/) (2026)
- [Benchmarking Foundation and Large Language Models for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2607.27856/) (2026)
- [Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs](https://mlsanity.com/paper/2608.17427/) (2026)
- [Cross-Modal MRI Ovary Segmentation in Endometriosis Using Unpaired TVUS Prototype Priors](https://mlsanity.com/paper/2608.18515/) (2026)
- [A Few Cases Are All You Need: An Empirical Study of Annotation-Efficient LoRA Fine-Tuning of MedSAM3](https://mlsanity.com/paper/2608.18731/) (2026)
- [Volumetric Radiology AI in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2608.20549/) (2026)
- [From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation](https://mlsanity.com/paper/2608.26856/) (2026)
- [SegBanana: Steering Unified Multimodal Models into Medical Segmenters](https://mlsanity.com/paper/2609.34235/) (2026)

## Abstract (arXiv)

Medical image segmentation is fundamental for biomedical discovery. Existing methods lack generalizability and demand extensive, time-consuming manual annotation for new clinical application. Here, we propose MedSAM-3, a text promptable medical segmentation model for medical image and video segmentation. By fine-tuning the Segment Anything Model (SAM) 3 architecture on medical images paired with semantic conceptual labels, our MedSAM-3 enables medical Promptable Concept Segmentation (PCS), allowing precise targeting of anatomical structures via open-vocabulary text descriptions rather than solely geometric prompts. We further introduce the MedSAM-3 Agent, a framework that integrates Multimodal Large Language Models (MLLMs) to perform complex reasoning and iterative refinement in an agent-in-the-loop workflow. Comprehensive experiments across diverse medical imaging modalities, including X-ray, MRI, Ultrasound, CT, and video, demonstrate that our approach significantly outperforms existing specialist and foundation models. We will release our code and model at https://github.com/Joey-S-Liu/MedSAM3.