# U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding

arXiv: https://arxiv.org/abs/2505.17779 · HTML: https://mlsanity.com/paper/2505.17779/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 74 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 32)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography](https://mlsanity.com/paper/1908.06948/) (2019)
- [Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data](https://mlsanity.com/paper/2308.02463/) (2023)
- [On the Tool Manipulation Capability of Open-source Large Language Models](https://mlsanity.com/paper/2305.16504/) (2023)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI](https://mlsanity.com/paper/2408.03361/) (2024)
- [Advancing Multimodal Medical Capabilities of Gemini](https://mlsanity.com/paper/2405.03162/) (2024)
- [A New Dataset and A Baseline Model for Breast Lesion Detection in Ultrasound Videos](https://mlsanity.com/paper/2207.00141/) (2022)
- [Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment](https://mlsanity.com/paper/2403.02738/) (2024)
- [Tracked 3D Ultrasound and Deep Neural Network-based Thyroid Segmentation reduce Interobserver Variability in Thyroid Volumetry](https://mlsanity.com/paper/2108.10118/) (2021)
- [MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis](https://mlsanity.com/paper/2407.04106/) (2024)
- [Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine](https://mlsanity.com/paper/2412.09278/) (2024)
- [The Open Kidney Ultrasound Data Set](https://mlsanity.com/paper/2206.06657/) (2022)
- [MI-SegNet: Mutual Information-Based US Segmentation for Unseen Domain Generalization](https://mlsanity.com/paper/2303.12649/) (2023)
- [Ultrasound Report Generation with Cross-Modality Feature Alignment via Unsupervised Guidance](https://mlsanity.com/paper/2406.00644/) (2024)
- [HoVer-Trans: Anatomy-aware HoVer-Transformer for ROI-free Breast Cancer Diagnosis in Ultrasound Images](https://mlsanity.com/paper/2205.08390/) (2022)
- [SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark](https://mlsanity.com/paper/2411.05521/) (2024)
- [MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video](https://mlsanity.com/paper/2408.03761/) (2024)
- [FPUS23: An Ultrasound Fetus Phantom Dataset with Deep Neural Network Evaluations for Fetus Orientations, Fetal Planes, and Anatomical Features](https://mlsanity.com/paper/2303.07852/) (2023)
- [Nerve Block Target Localization and Needle Guidance for Autonomous Robotic Ultrasound Guided Regional Anesthesia](https://mlsanity.com/paper/2308.03717/) (2023)

## Built on by (most cited first, 10 of 10)

- [Ultrasound-CLIP: Semantic-Aware Contrastive Pre-training for Ultrasound Image-Text Understanding](https://mlsanity.com/paper/2604.01749/) (2026)
- [Grounding DINO-US-SAM: Text-Prompted Multi-Organ Segmentation in Ultrasound with LoRA-Tuned Vision-Language Models](https://mlsanity.com/paper/2506.23903/) (2025)
- [MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models](https://mlsanity.com/paper/2606.06696/) (2026)
- [Multimodal Carotid Risk Stratification with Large Vision-Language Models: Benchmarking, Fine-Tuning, and Clinical Insights](https://mlsanity.com/paper/2510.02922/) (2025)
- [SpineBench: Benchmarking Multimodal LLMs for Spinal Pathology Analysis](https://mlsanity.com/paper/2510.12267/) (2025)
- [When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasound](https://mlsanity.com/paper/2603.21047/) (2026)
- [ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding](https://mlsanity.com/paper/2604.10916/) (2026)
- [Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming](https://mlsanity.com/paper/2605.21652/) (2026)
- [UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound](https://mlsanity.com/paper/2609.28610/) (2026)
- [A Concept-informed Interpretable Artificial Intelligence Framework for Breast Ultrasound Diagnosis](https://mlsanity.com/paper/2610.16128/) (2026)

## Abstract (arXiv)

Ultrasound is a widely-used imaging modality critical to global healthcare, yet its interpretation remains challenging due to its varying image quality on operators, noises, and anatomical structures. Although large vision-language models (LVLMs) have demonstrated impressive multimodal capabilities across natural and medical domains, their performance on ultrasound remains largely unexplored. We introduce U2-BENCH, the first comprehensive benchmark to evaluate LVLMs on ultrasound understanding across classification, detection, regression, and text generation tasks. U2-BENCH aggregates 7,241 cases spanning 15 anatomical regions and defines 8 clinically inspired tasks, such as diagnosis, view recognition, lesion localization, clinical value estimation, and report generation, across 50 ultrasound application scenarios. We evaluate 23 state-of-the-art LVLMs, both open- and closed-source, general-purpose and medical-specific. Our results reveal strong performance on image-level classification, but persistent challenges in spatial reasoning and clinical language generation. U2-BENCH establishes a rigorous and unified testbed to assess and accelerate LVLM research in the uniquely multimodal domain of medical ultrasound imaging.