# Ultrasound-CLIP: Semantic-Aware Contrastive Pre-training for Ultrasound Image-Text Understanding

arXiv: https://arxiv.org/abs/2604.01749 · HTML: https://mlsanity.com/paper/2604.01749/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 40 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 7 of 7)

- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine](https://mlsanity.com/paper/2408.02900/) (2024)
- [UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities](https://mlsanity.com/paper/2412.10372/) (2024)
- [FetalCLIP: A Visual-Language Foundation Model for Fetal Ultrasound Image Analysis](https://mlsanity.com/paper/2502.14807/) (2025)
- [U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding](https://mlsanity.com/paper/2505.17779/) (2025)
- [MMCLIP: Cross-modal Attention Masked Modelling for Medical Language-Image Pre-Training](https://mlsanity.com/paper/2407.19546/) (2024)
- [HAIBU-ReMUD: Reasoning Multimodal Ultrasound Dataset and Model Bridging to General Specific Domains](https://mlsanity.com/paper/2506.07837/) (2025)

## Built on by (most cited first, 4 of 4)

- [FADA: Accessible fetal ultrasound interpretation and annotation with a selectively distilled unified vision-language model](https://mlsanity.com/paper/2606.11106/) (2026)
- [Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports](https://mlsanity.com/paper/2607.01908/) (2026)
- [Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening](https://mlsanity.com/paper/2608.14763/) (2026)
- [UltraPIPS: Improving model perception in B-mode ultrasound with foundation models](https://mlsanity.com/paper/2608.26033/) (2026)

## Abstract (arXiv)

Ultrasound imaging is widely used in clinical diagnostics due to its real-time capability and radiation-free nature. However, existing vision-language pre-training models, such as CLIP, are primarily designed for other modalities, and are difficult to directly apply to ultrasound data, which exhibit heterogeneous anatomical structures and diverse diagnostic attributes. To bridge this gap, we construct US-365K, a large-scale ultrasound image-text dataset containing 365k paired samples across 52 anatomical categories. We establish Ultrasonographic Diagnostic Taxonomy (UDT) containing two hierarchical knowledge frameworks. Ultrasonographic Hierarchical Anatomical Taxonomy standardizes anatomical organization, and Ultrasonographic Diagnostic Attribute Framework formalizes nine diagnostic dimensions, including body system, organ, diagnosis, shape, margins, echogenicity, internal characteristics, posterior acoustic phenomena, and vascularity. Building upon these foundations, we propose Ultrasound-CLIP, a semantic-aware contrastive learning framework that introduces semantic soft labels and semantic loss to refine sample discrimination. Moreover, we construct a heterogeneous graph modality derived from UDAF's textual representations, enabling structured reasoning over lesion-attribute relations. Extensive experiments with patient-level data splitting demonstrate that our approach achieves state-of-the-art performance on classification and retrieval benchmarks, while also delivering strong generalization to zero-shot, linear probing, and fine-tuning tasks.