# Grounding DINO-US-SAM: Text-Prompted Multi-Organ Segmentation in Ultrasound with LoRA-Tuned Vision-Language Models

arXiv: https://arxiv.org/abs/2506.23903 · HTML: https://mlsanity.com/paper/2506.23903/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 63 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once](https://mlsanity.com/paper/2405.12971/) (2024)
- [UniverSeg: Universal Medical Image Segmentation](https://mlsanity.com/paper/2304.06131/) (2023)
- [AAU-net: An Adaptive Attention U-net for Breast Lesions Segmentation in Ultrasound Images](https://mlsanity.com/paper/2204.12077/) (2022)
- [Beyond Adapting SAM: Towards End-to-End Ultrasound Image Segmentation via Auto Prompting](https://mlsanity.com/paper/2309.06824/) (2023)
- [MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentation](https://mlsanity.com/paper/2409.19483/) (2024)
- [MedCLIP-SAM: Bridging Text and Image Towards Universal Medical Image Segmentation](https://mlsanity.com/paper/2403.20253/) (2024)
- [BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models](https://mlsanity.com/paper/2411.15232/) (2024)
- [MicroSegNet: A Deep Learning Approach for Prostate Segmentation on Micro-Ultrasound Images](https://mlsanity.com/paper/2305.19956/) (2023)
- [UltraSam: A Foundation Model for Ultrasound using Large Open-Access Segmentation Datasets](https://mlsanity.com/paper/2411.16222/) (2024)
- [Interpreting Object-level Foundation Models via Visual Precision Search](https://mlsanity.com/paper/2411.16198/) (2024)
- [The Open Kidney Ultrasound Data Set](https://mlsanity.com/paper/2206.06657/) (2022)
- [CC-SAM: SAM with Cross-feature Attention and Context for Ultrasound Image Segmentation](https://mlsanity.com/paper/2408.00181/) (2024)
- [U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding](https://mlsanity.com/paper/2505.17779/) (2025)
- [Intrapartum Ultrasound Image Segmentation of Pubic Symphysis and Fetal Head Using Dual Student-Teacher Framework with CNN-ViT Collaborative Learning](https://mlsanity.com/paper/2409.06928/) (2024)
- [Less is More: Adaptive Curriculum Learning for Thyroid Nodule Diagnosis](https://mlsanity.com/paper/2207.00807/) (2022)
- [Algorithm guided outlining of 105 pancreatic cancer liver metastases in Ultrasound](https://mlsanity.com/paper/1710.02984/) (2017)
- [ClickSAM: Fine-tuning Segment Anything Model using click prompts for ultrasound image segmentation](https://mlsanity.com/paper/2402.05902/) (2024)
- [Striving for Simplicity: Simple Yet Effective Prior-Aware Pseudo-Labeling for Semi-Supervised Ultrasound Image Segmentation](https://mlsanity.com/paper/2503.13987/) (2025)
- [LightBTSeg: A lightweight breast tumor segmentation model using ultrasound images via dual-path joint knowledge distillation](https://mlsanity.com/paper/2311.11086/) (2023)
- [Deep Spectral Methods for Unsupervised Ultrasound Image Interpretation](https://mlsanity.com/paper/2408.02043/) (2024)

## Built on by (most cited first, 3 of 3)

- [LoGSAM: Parameter-Efficient Cross-Modal Grounding for MRI Segmentation](https://mlsanity.com/paper/2603.17576/) (2026)
- [Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning](https://mlsanity.com/paper/2605.26292/) (2026)
- [ThyCLIPNet: A BiomedCLIP-Guided Lightweight Attention-Enhanced DeepLabV3+ Framework for Robust Thyroid Nodule Segmentation](https://mlsanity.com/paper/2610.04743/) (2026)

## Abstract (arXiv)

Accurate and generalizable object segmentation in ultrasound imaging remains a significant challenge due to anatomical variability, diverse imaging protocols, and limited annotated data. In this study, we propose a prompt-driven vision-language model (VLM) that integrates Grounding DINO with SAM2 (Segment Anything Model2) to enable object segmentation across multiple ultrasound organs. A total of 18 public ultrasound datasets, encompassing the breast, thyroid, liver, prostate, kidney, and paraspinal muscle, were utilized. These datasets were divided into 15 for fine-tuning and validation of Grounding DINO using Low Rank Adaptation (LoRA) to the ultrasound domain, and 3 were held out entirely for testing to evaluate performance in unseen distributions. Comprehensive experiments demonstrate that our approach outperforms state-of-the-art segmentation methods, including UniverSeg, MedSAM, MedCLIP-SAM, BiomedParse, and SAMUS on most seen datasets while maintaining strong performance on unseen datasets without additional fine-tuning. These results underscore the promise of VLMs in scalable and robust ultrasound image analysis, reducing dependence on large, organ-specific annotated datasets. We will publish our code on code.sonography.ai after acceptance.