# UltraSam: A Foundation Model for Ultrasound using Large Open-Access Segmentation Datasets

arXiv: https://arxiv.org/abs/2411.16222 · HTML: https://mlsanity.com/paper/2411.16222/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 13 of 13)

- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography](https://mlsanity.com/paper/1908.06948/) (2019)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [USFM: A Universal Ultrasound Foundation Model Generalized to Tasks and Organs towards Label Efficient Image Analysis](https://mlsanity.com/paper/2401.00153/) (2023)
- [Tracked 3D Ultrasound and Deep Neural Network-based Thyroid Segmentation reduce Interobserver Variability in Thyroid Volumetry](https://mlsanity.com/paper/2108.10118/) (2021)
- [MicroSegNet: A Deep Learning Approach for Prostate Segmentation on Micro-Ultrasound Images](https://mlsanity.com/paper/2305.19956/) (2023)
- [GraphEcho: Graph-Driven Unsupervised Domain Adaptation for Echocardiogram Video Segmentation](https://mlsanity.com/paper/2309.11145/) (2023)
- [The Open Kidney Ultrasound Data Set](https://mlsanity.com/paper/2206.06657/) (2022)
- [EchoCP: An Echocardiography Dataset in Contrast Transthoracic Echocardiography for Patent Foramen Ovale Diagnosis](https://mlsanity.com/paper/2105.08267/) (2021)
- [Algorithm guided outlining of 105 pancreatic cancer liver metastases in Ultrasound](https://mlsanity.com/paper/1710.02984/) (2017)
- [Ultrasound SAM Adapter: Adapting SAM for Breast Lesion Segmentation in Ultrasound Images](https://mlsanity.com/paper/2404.14837/) (2024)

## Built on by (most cited first, 24 of 24)

- [USF-MAE: Ultrasound Self-Supervised Foundation Model with Masked Autoencoding](https://mlsanity.com/paper/2510.22990/) (2025)
- [A Fully Open and Generalizable Foundation Model for Ultrasound Clinical Applications](https://mlsanity.com/paper/2509.11752/) (2025)
- [Adapting Vision-Language Foundation Model for Next Generation Medical Ultrasound Image Analysis](https://mlsanity.com/paper/2506.08849/) (2025)
- [Grounding DINO-US-SAM: Text-Prompted Multi-Organ Segmentation in Ultrasound with LoRA-Tuned Vision-Language Models](https://mlsanity.com/paper/2506.23903/) (2025)
- [FreqDINO: Frequency-Guided Adaptation for Generalized Boundary-Aware Ultrasound Image Segmentation](https://mlsanity.com/paper/2512.11335/) (2025)
- [UMind-VL: A Generalist Ultrasound Vision-Language Model for Unified Grounded Perception and Comprehensive Interpretation](https://mlsanity.com/paper/2511.22256/) (2025)
- [Improved cystic hygroma detection from prenatal imaging using ultrasound-specific self-supervised representation learning](https://mlsanity.com/paper/2512.22730/) (2025)
- [US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound](https://mlsanity.com/paper/2602.19322/) (2026)
- [COME: Dual Structure-Semantic Learning with Collaborative MoE for Universal Lesion Detection Across Heterogeneous Ultrasound Datasets](https://mlsanity.com/paper/2508.09886/) (2025)
- [Automated Classification of First-Trimester Fetal Heart Views Using Ultrasound-Specific Self-Supervised Learning](https://mlsanity.com/paper/2512.24492/) (2025)
- [A texture-based framework for foundational ultrasound models](https://mlsanity.com/paper/2602.01444/) (2026)
- [Vision Foundation Models in Radiology: A Scoping Review of Data, Methodology, Evaluation and Clinical Translation](https://mlsanity.com/paper/2607.07219/) (2026)
- [UltraSAM3: A Concept-Driven Foundation Model for Universal Ultrasound Image Segmentation](https://mlsanity.com/paper/2607.29200/) (2026)
- [AURORA: Adaptive Unified Representation for Robust Ultrasound Analysis](https://mlsanity.com/paper/2603.19364/) (2026)
- [Benchmarking Ultrasound Foundation Models for Fetal Plane Classification](https://mlsanity.com/paper/2605.27796/) (2026)
- [FADA: Accessible fetal ultrasound interpretation and annotation with a selectively distilled unified vision-language model](https://mlsanity.com/paper/2606.11106/) (2026)
- [Analyzing Image Encoder Choices and Graph Homophily in GCN Frameworks for Breast Ultrasound Classification](https://mlsanity.com/paper/2607.12054/) (2026)
- [Learning from Complementary Ultrasound Representations for Liver Disease Classification](https://mlsanity.com/paper/2607.12062/) (2026)
- [FedCC: A Low-Resource Federated Adaptation of Foundation Models for Robust Corpus Callosum localization in Fetal Ultrasound Images](https://mlsanity.com/paper/2607.18283/) (2026)
- [UltraPIPS: Improving model perception in B-mode ultrasound with foundation models](https://mlsanity.com/paper/2608.26033/) (2026)
- [FreqDINO++: A Frequency-Guided Multi-Task Routing Vision Foundation Model for Universal Ultrasound Analysis](https://mlsanity.com/paper/2609.20340/) (2026)
- [ORION-CMR: On-scanner Reporting with Integrated Foundation Model for End-to-End Cardiac MRI Analysis and Interpretation](https://mlsanity.com/paper/2609.23950/) (2026)
- [UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound](https://mlsanity.com/paper/2609.28610/) (2026)
- [A Concept-informed Interpretable Artificial Intelligence Framework for Breast Ultrasound Diagnosis](https://mlsanity.com/paper/2610.16128/) (2026)

## Abstract (arXiv)

Purpose: Automated ultrasound image analysis is challenging due to anatomical complexity and limited annotated data. To tackle this, we take a data-centric approach, assembling the largest public ultrasound segmentation dataset and training a versatile visual foundation model tailored for ultrasound. Methods: We compile US-43d, a large-scale collection of 43 open-access ultrasound datasets with over 280,000 images and segmentation masks for more than 50 anatomical structures. We then introduce UltraSam, an adaptation of the Segment Anything Model (SAM) that is trained on US-43d and supports both point- and box-prompts. Finally, we introduce a new use case for SAM-style models by using UltraSam as a model initialization that can be fine-tuned for various downstream analysis tasks, demonstrating UltraSam's foundational capabilities. Results: UltraSam achieves vastly improved performance over existing SAM-style models for prompt-based segmentation on three diverse public datasets. Moreover, an UltraSam-initialized Vision Transformer surpasses ImageNet-, SAM-, and MedSAM-initialized models in various downstream segmentation and classification tasks, highlighting UltraSam's effectiveness as a foundation model. Conclusion: We compile US-43d, a large-scale unified ultrasound dataset, and introduce UltraSam, a powerful multi-purpose SAM-style model for ultrasound images. We release our code and pretrained models at https://github.com/CAMMA-public/UltraSam and invite the community to further this effort by contributing high-quality datasets.