# ProMISe: Promptable Medical Image Segmentation using SAM

arXiv: https://arxiv.org/abs/2403.04164 · HTML: https://mlsanity.com/paper/2403.04164/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 7 Mar 2024.

## Builds on (most cited first, 23 of 23)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images](https://mlsanity.com/paper/1612.00799/) (2016)
- [ResUNet++: An Advanced Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1911.07067/) (2019)
- [Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1605.01397/) (2016)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [SAM-Med2D](https://mlsanity.com/paper/2308.16184/) (2023)
- [SAM on Medical Images: A Comprehensive Study on Three Prompt Modes](https://mlsanity.com/paper/2305.00035/) (2023)
- [AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encoder](https://mlsanity.com/paper/2306.06370/) (2023)
- [Polyp-SAM: Transfer SAM for Polyp Segmentation](https://mlsanity.com/paper/2305.00293/) (2023)
- [Can SAM Segment Polyps?](https://mlsanity.com/paper/2304.07583/) (2023)
- [Stepwise Feature Fusion: Local Guides Global](https://mlsanity.com/paper/2203.03635/) (2022)
- [AdaptiveSAM: Towards Efficient Tuning of SAM for Surgical Scene Segmentation](https://mlsanity.com/paper/2308.03726/) (2023)
- [DuAT: Dual-Aggregation Transformer Network for Medical Image Segmentation](https://mlsanity.com/paper/2212.11677/) (2022)

## Built on by (most cited first, 3 of 3)

- [MedSAM-U: Uncertainty-Guided Auto Multi-Prompt Adaptation for Reliable MedSAM](https://mlsanity.com/paper/2409.00924/) (2024)
- [Promptable cancer segmentation using minimal expert-curated data](https://mlsanity.com/paper/2505.17915/) (2025)
- [SAMPO-Path: Segmentation Intent-Aligned Preference Optimization for Pathology Foundation Model Segmentation](https://mlsanity.com/paper/2508.02464/) (2025)

## Abstract (arXiv)

With the proposal of the Segment Anything Model (SAM), fine-tuning SAM for medical image segmentation (MIS) has become popular. However, due to the large size of the SAM model and the significant domain gap between natural and medical images, fine-tuning-based strategies are costly with potential risk of instability, feature damage and catastrophic forgetting. Furthermore, some methods of transferring SAM to a domain-specific MIS through fine-tuning strategies disable the model's prompting capability, severely limiting its utilization scenarios. In this paper, we propose an Auto-Prompting Module (APM), which provides SAM-based foundation model with Euclidean adaptive prompts in the target domain. Our experiments demonstrate that such adaptive prompts significantly improve SAM's non-fine-tuned performance in MIS. In addition, we propose a novel non-invasive method called Incremental Pattern Shifting (IPS) to adapt SAM to specific medical domains. Experimental results show that the IPS enables SAM to achieve state-of-the-art or competitive performance in MIS without the need for fine-tuning. By coupling these two methods, we propose ProMISe, an end-to-end non-fine-tuned framework for Promptable Medical Image Segmentation. Our experiments demonstrate that both using our methods individually or in combination achieves satisfactory performance in low-cost pattern shifting, with all of SAM's parameters frozen.