# Semantic Segmentation Refiner for Ultrasound Applications with Zero-Shot Foundation Models

arXiv: https://arxiv.org/abs/2404.16325 · HTML: https://mlsanity.com/paper/2404.16325/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [SAM-Med2D](https://mlsanity.com/paper/2308.16184/) (2023)
- [SAM on Medical Images: A Comprehensive Study on Three Prompt Modes](https://mlsanity.com/paper/2305.00035/) (2023)
- [How to Efficiently Adapt Large Segmentation Model(SAM) to Medical Images](https://mlsanity.com/paper/2306.13731/) (2023)
- [Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets](https://mlsanity.com/paper/2304.09324/) (2023)
- [SAM-PARSER: Fine-tuning SAM Efficiently by Parameter Space Reconstruction](https://mlsanity.com/paper/2308.14604/) (2023)
- [Cheap Lunch for Medical Image Segmentation by Fine-tuning SAM on Few Exemplars](https://mlsanity.com/paper/2308.14133/) (2023)
- [Simultaneous Segmentation and Classification of Bone Surfaces from Ultrasound Using a Multi-feature Guided CNN](https://mlsanity.com/paper/1806.09766/) (2018)
- [One-shot Localization and Segmentation of Medical Images with Foundation Models](https://mlsanity.com/paper/2310.18642/) (2023)
- [Contrastive Attraction and Contrastive Repulsion for Representation Learning](https://mlsanity.com/paper/2105.03746/) (2021)
- [SAM-OCTA: Prompting Segment-Anything for OCTA Image Segmentation](https://mlsanity.com/paper/2310.07183/) (2023)

## Built on by (most cited first, 2 of 2)

- [U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding](https://mlsanity.com/paper/2505.17779/) (2025)
- [CSG: A Context-Semantic Guided Diffusion Approach in De Novo Musculoskeletal Ultrasound Image Generation](https://mlsanity.com/paper/2412.05833/) (2024)

## Abstract (arXiv)

Despite the remarkable success of deep learning in medical imaging analysis, medical image segmentation remains challenging due to the scarcity of high-quality labeled images for supervision. Further, the significant domain gap between natural and medical images in general and ultrasound images in particular hinders fine-tuning models trained on natural images to the task at hand. In this work, we address the performance degradation of segmentation models in low-data regimes and propose a prompt-less segmentation method harnessing the ability of segmentation foundation models to segment abstract shapes. We do that via our novel prompt point generation algorithm which uses coarse semantic segmentation masks as input and a zero-shot prompt-able foundation model as an optimization target. We demonstrate our method on a segmentation findings task (pathologic anomalies) in ultrasound images. Our method's advantages are brought to light in varying degrees of low-data regime experiments on a small-scale musculoskeletal ultrasound images dataset, yielding a larger performance gain as the training set size decreases.