# Diffusion-empowered AutoPrompt MedSAM

arXiv: https://arxiv.org/abs/2502.06817 · HTML: https://mlsanity.com/paper/2502.06817/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Builds on (most cited first, 30 of 32)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [Medical Image Segmentation Review: The success of U-Net](https://mlsanity.com/paper/2211.14830/) (2022)
- [Wider or Deeper: Revisiting the ResNet Model for Visual Recognition](https://mlsanity.com/paper/1611.10080/) (2016)
- [nnFormer: Interleaved Transformer for Volumetric Segmentation](https://mlsanity.com/paper/2109.03201/) (2021)
- [SAM-Med2D](https://mlsanity.com/paper/2308.16184/) (2023)
- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions](https://mlsanity.com/paper/2401.03495/) (2024)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [How to Efficiently Adapt Large Segmentation Model(SAM) to Medical Images](https://mlsanity.com/paper/2306.13731/) (2023)
- [Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding](https://mlsanity.com/paper/2403.18271/) (2024)
- [A Data-scalable Transformer for Medical Image Segmentation: Architecture, Model Efficiency, and Benchmark](https://mlsanity.com/paper/2203.00131/) (2022)
- [AdaptiveSAM: Towards Efficient Tuning of SAM for Surgical Scene Segmentation](https://mlsanity.com/paper/2308.03726/) (2023)
- [Diffusion Models for Counterfactual Generation and Anomaly Detection in Brain Images](https://mlsanity.com/paper/2308.02062/) (2023)
- [Synthetic CT Generation from MRI using 3D Transformer-based Denoising Diffusion Model](https://mlsanity.com/paper/2305.19467/) (2023)
- [Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomy](https://mlsanity.com/paper/2311.13964/) (2023)
- [Brain tumor segmentation using synthetic MR images -- A comparison of GANs and diffusion models](https://mlsanity.com/paper/2306.02986/) (2023)
- [AutoProSAM: Automated Prompting SAM for 3D Multi-Organ Segmentation](https://mlsanity.com/paper/2308.14936/) (2023)
- [MedSAM-U: Uncertainty-Guided Auto Multi-Prompt Adaptation for Reliable MedSAM](https://mlsanity.com/paper/2409.00924/) (2024)
- [UU-Mamba: Uncertainty-aware U-Mamba for Cardiac Image Segmentation](https://mlsanity.com/paper/2405.17496/) (2024)
- [Robustly Optimized Deep Feature Decoupling Network for Fatty Liver Diseases Detection](https://mlsanity.com/paper/2406.17338/) (2024)

## Built on by (most cited first, 10 of 10)

- [Improving Generalization of Medical Image Registration Foundation Model](https://mlsanity.com/paper/2505.06527/) (2025)
- [Robust AI-Generated Face Detection with Imbalanced Data](https://mlsanity.com/paper/2505.02182/) (2025)
- [MedChat: A Multi-Agent Framework for Multimodal Diagnosis with Large Language Models](https://mlsanity.com/paper/2506.07400/) (2025)
- [Robust Fairness Vision-Language Learning for Medical Image Analysis](https://mlsanity.com/paper/2505.03153/) (2025)
- [Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges](https://mlsanity.com/paper/2507.09562/) (2025)
- [Interpretable Unsupervised Deformable Image Registration via Confidence-bound Multi-Hop Visual Reasoning](https://mlsanity.com/paper/2602.00211/) (2026)
- [Attention-Driven Framework for Non-Rigid Medical Image Registration](https://mlsanity.com/paper/2602.07088/) (2026)
- [Robust Multi-Source Covid-19 Detection in CT Images](https://mlsanity.com/paper/2604.03320/) (2026)
- [Weight Group-wise Post-Training Quantization for Medical Foundation Model](https://mlsanity.com/paper/2604.07674/) (2026)
- [Improving Cross-Site Whole-Heart Segmentation](https://mlsanity.com/paper/2608.25109/) (2026)

## Abstract (arXiv)

MedSAM, a medical foundation model derived from the SAM architecture, has demonstrated notable success across diverse medical domains. However, its clinical application faces two major challenges: the dependency on labor-intensive manual prompt generation, which imposes a significant burden on clinicians, and the absence of semantic labeling in the generated segmentation masks for organs or lesions, limiting its practicality for non-expert users. To address these limitations, we propose AutoMedSAM, an end-to-end framework derived from SAM, designed to enhance usability and segmentation performance. AutoMedSAM retains MedSAM's image encoder and mask decoder structure while introducing a novel diffusion-based class prompt encoder. The diffusion-based encoder employs a dual-decoder structure to collaboratively generate prompt embeddings guided by sparse and dense prompt definitions. These embeddings enhance the model's ability to understand and process clinical imagery autonomously. With this encoder, AutoMedSAM leverages class prompts to embed semantic information into the model's predictions, transforming MedSAM's semi-automated pipeline into a fully automated workflow. Furthermore, AutoMedSAM employs an uncertainty-aware joint optimization strategy during training to effectively inherit MedSAM's pre-trained knowledge while improving generalization by integrating multiple loss functions. Experimental results across diverse datasets demonstrate that AutoMedSAM achieves superior performance while broadening its applicability to both clinical settings and non-expert users. Code is available at https://github.com/HP-ML/AutoPromptMedSAM.git.