# Beyond Adapting SAM: Towards End-to-End Ultrasound Image Segmentation via Auto Prompting

arXiv: https://arxiv.org/abs/2309.06824 · HTML: https://mlsanity.com/paper/2309.06824/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation](https://mlsanity.com/paper/2102.08005/) (2021)
- [Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography](https://mlsanity.com/paper/1908.06948/) (2019)
- [CE-Net: Context Encoder Network for 2D Medical Image Segmentation](https://mlsanity.com/paper/1903.02740/) (2019)
- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [CA-Net: Comprehensive Attention Convolutional Neural Networks for Explainable Medical Image Segmentation](https://mlsanity.com/paper/2009.10549/) (2020)
- [AAU-net: An Adaptive Attention U-net for Breast Lesions Segmentation in Ultrasound Images](https://mlsanity.com/paper/2204.12077/) (2022)
- [AdaptiveSAM: Towards Efficient Tuning of SAM for Surgical Scene Segmentation](https://mlsanity.com/paper/2308.03726/) (2023)
- [Polyp-SAM++: Can A Text Guided SAM Perform Better for Polyp Segmentation?](https://mlsanity.com/paper/2308.06623/) (2023)
- [Left Ventricular Wall Motion Estimation by Active Polynomials for Acute Myocardial Infarction Detection](https://mlsanity.com/paper/2008.04615/) (2020)

## Built on by (most cited first, 30 of 41)

- [Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions](https://mlsanity.com/paper/2401.03495/) (2024)
- [ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Image](https://mlsanity.com/paper/2312.07381/) (2023)
- [Foundation Models for Biomedical Image Segmentation: A Survey](https://mlsanity.com/paper/2401.07654/) (2024)
- [UN-SAM: Universal Prompt-Free Segmentation for Generalized Nuclei Images](https://mlsanity.com/paper/2402.16663/) (2024)
- [Unleashing the Power of Prompt-driven Nucleus Instance Segmentation](https://mlsanity.com/paper/2311.15939/) (2023)
- [EchoWorld: Learning Motion-Aware World Models for Echocardiography Probe Guidance](https://mlsanity.com/paper/2504.13065/) (2025)
- [ASPS: Augmented Segment Anything Model for Polyp Segmentation](https://mlsanity.com/paper/2407.00718/) (2024)
- [Adapting Vision Foundation Models for Real-time Ultrasound Image Segmentation](https://mlsanity.com/paper/2503.24368/) (2025)
- [SAMCT: Segment Any CT Allowing Labor-Free Task-Indicator Prompts](https://mlsanity.com/paper/2403.13258/) (2024)
- [TextSAM-EUS: Text Prompt Learning for SAM to Accurately Segment Pancreatic Tumor in Endoscopic Ultrasound](https://mlsanity.com/paper/2507.18082/) (2025)
- [UniUltra: Interactive Parameter-Efficient SAM2 for Universal Ultrasound Segmentation](https://mlsanity.com/paper/2511.15771/) (2025)
- [Ultrasound SAM Adapter: Adapting SAM for Breast Lesion Segmentation in Ultrasound Images](https://mlsanity.com/paper/2404.14837/) (2024)
- [Leveraging Segment Anything Model for Source-Free Domain Adaptation via Dual Feature Guided Auto-Prompting](https://mlsanity.com/paper/2505.08527/) (2025)
- [Grounding DINO-US-SAM: Text-Prompted Multi-Organ Segmentation in Ultrasound with LoRA-Tuned Vision-Language Models](https://mlsanity.com/paper/2506.23903/) (2025)
- [Anatomy-Guided Representation Learning Using a Transformer-Based Network for Thyroid Nodule Segmentation in Ultrasound Images](https://mlsanity.com/paper/2512.12662/) (2025)
- [US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound](https://mlsanity.com/paper/2602.19322/) (2026)
- [Leveraging SAM for Single-Source Domain Generalization in Medical Image Segmentation](https://mlsanity.com/paper/2401.02076/) (2024)
- [Reinforcement Learning for Unsupervised Domain Adaptation in Spatio-Temporal Echocardiography Segmentation](https://mlsanity.com/paper/2510.14244/) (2025)
- [Glioma C6: A Novel Dataset for Training and Benchmarking Cell Segmentation](https://mlsanity.com/paper/2511.07286/) (2025)
- [UI-Styler: Ultrasound Image Style Transfer with Class-Aware Prompts for Cross-Device Diagnosis Using a Frozen Black-Box Inference Network](https://mlsanity.com/paper/2511.17155/) (2025)
- [XSeg: A Large-scale X-ray Contraband Segmentation Benchmark For Real-World Security Screening](https://mlsanity.com/paper/2604.03706/) (2026)
- [BOTM: Echocardiography Segmentation via Bi-directional Optimal Token Matching](https://mlsanity.com/paper/2505.18052/) (2025)
- [Think as Cardiac Sonographers: Marrying SAM with Left Ventricular Indicators Measurements According to Clinical Guidelines](https://mlsanity.com/paper/2508.08566/) (2025)
- [TRUST: Token-dRiven Ultrasound Style Transfer for Cross-Device Adaptation](https://mlsanity.com/paper/2509.00508/) (2025)
- [EMA-SAM: Exponential Moving-average for SAM-based PTMC Segmentation](https://mlsanity.com/paper/2510.18213/) (2025)
- [Externally Validated Breast Ultrasound Segmentation via Multi-task Learning with BI-RADS-Consistent Morphological Priors](https://mlsanity.com/paper/2511.15968/) (2025)
- [Semantic-aware Random Convolution and Source Matching for Domain Generalization in Medical Image Segmentation](https://mlsanity.com/paper/2512.01510/) (2025)
- [OFL-SAM2: Prompt SAM2 with Online Few-shot Learner for Efficient Medical Image Segmentation](https://mlsanity.com/paper/2512.24861/) (2025)
- [Point Tracking as a Temporal Cue for Robust Myocardial Segmentation in Echocardiography Videos](https://mlsanity.com/paper/2601.09207/) (2026)
- [FetalAgents: A Multi-Agent System for Fetal Ultrasound Image and Video Analysis](https://mlsanity.com/paper/2603.09733/) (2026)

## Abstract (arXiv)

End-to-end medical image segmentation is of great value for computer-aided diagnosis dominated by task-specific models, usually suffering from poor generalization. With recent breakthroughs brought by the segment anything model (SAM) for universal image segmentation, extensive efforts have been made to adapt SAM for medical imaging but still encounter two major issues: 1) severe performance degradation and limited generalization without proper adaptation, and 2) semi-automatic segmentation relying on accurate manual prompts for interaction. In this work, we propose SAMUS as a universal model tailored for ultrasound image segmentation and further enable it to work in an end-to-end manner denoted as AutoSAMUS. Specifically, in SAMUS, a parallel CNN branch is introduced to supplement local information through cross-branch attention, and a feature adapter and a position adapter are jointly used to adapt SAM from natural to ultrasound domains while reducing training complexity. AutoSAMUS is realized by introducing an auto prompt generator (APG) to replace the manual prompt encoder of SAMUS to automatically generate prompt embeddings. A comprehensive ultrasound dataset, comprising about 30k images and 69k masks and covering six object categories, is collected for verification. Extensive comparison experiments demonstrate the superiority of SAMUS and AutoSAMUS against the state-of-the-art task-specific and SAM-based foundation models. We believe the auto-prompted SAM-based model has the potential to become a new paradigm for end-to-end medical image segmentation and deserves more exploration. Code and data are available at https://github.com/xianlin7/SAMUS.