mlsanityData to 14 Oct 2026

U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding

arXiv 2505.17779 ↗First posted 23 May 2025cs.CV cs.LGcited by 10

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Ultrasound is a widely-used imaging modality critical to global healthcare, yet its interpretation remains challenging due to its varying image quality on operators, noises, and anatomical structures. Although large vision-language models (LVLMs) have demonstrated impressive multimodal capabilities across natural and medical domains, their performance on ultrasound remains largely unexplored. We introduce U2-BENCH, the first comprehensive benchmark to evaluate LVLMs on ultrasound understanding across classification, detection, regression, and text generation tasks. U2-BENCH aggregates 7,241 cases spanning 15 anatomical regions and defines 8 clinically inspired tasks, such as diagnosis, view recognition, lesion localization, clinical value estimation, and report generation, across 50 ultrasound application scenarios. We evaluate 23 state-of-the-art LVLMs, both open- and closed-source, general-purpose and medical-specific. Our results reveal strong performance on image-level classification, but persistent challenges in spatial reasoning and clinical language generation. U2-BENCH establishes a rigorous and unified testbed to assess and accelerate LVLM research in the uniquely multimodal domain of medical ultrasound imaging.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 74 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

32 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine12 Dec
  2. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI6 Aug

Other references · 30

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  9. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  10. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  11. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  12. 2019Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiographycited by 244
18 more
  1. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  2. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  3. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 115
  4. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  5. 2022A New Dataset and A Baseline Model for Breast Lesion Detection in Ultrasound Videoscited by 32
  6. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 30
  7. 2021Tracked 3D Ultrasound and Deep Neural Network-based Thyroid Segmentation reduce Interobserver Variability in Thyroid Volumetrycited by 29
  8. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosiscited by 28
  9. 2022The Open Kidney Ultrasound Data Setcited by 18
  10. 2023MI-SegNet: Mutual Information-Based US Segmentation for Unseen Domain Generalizationcited by 18
  11. 2024Ultrasound Report Generation with Cross-Modality Feature Alignment via Unsupervised Guidancecited by 17
  12. 2022HoVer-Trans: Anatomy-aware HoVer-Transformer for ROI-free Breast Cancer Diagnosis in Ultrasound Imagescited by 15
  13. 2024SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmarkcited by 14
  14. 2024MMSummary: Multimodal Summary Generation for Fetal Ultrasound Videocited by 9
  15. 2023FPUS23: An Ultrasound Fetus Phantom Dataset with Deep Neural Network Evaluations for Fetus Orientations, Fetal Planes, and Anatomical Featurescited by 8
  16. 2023Nerve Block Target Localization and Needle Guidance for Autonomous Robotic Ultrasound Guided Regional Anesthesiacited by 3
  17. 2024Leveraging AI for Automatic Classification of PCOS Using Ultrasound Imagingcited by 3
  18. 2024Semantic Segmentation Refiner for Ultrasound Applications with Zero-Shot Foundation Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026Ultrasound-CLIP: Semantic-Aware Contrastive Pre-training for Ultrasound Image-Text Understandingcited by 4
  2. 2025Grounding DINO-US-SAM: Text-Prompted Multi-Organ Segmentation in Ultrasound with LoRA-Tuned Vision-Language Modelscited by 3
  3. 2026MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Modelscited by 3
  4. 2025Multimodal Carotid Risk Stratification with Large Vision-Language Models: Benchmarking, Fine-Tuning, and Clinical Insightscited by 0
  5. 2025SpineBench: Benchmarking Multimodal LLMs for Spinal Pathology Analysiscited by 0
  6. 2026When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasoundcited by 0
  7. 2026ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understandingcited by 0
  8. 2026Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zoomingcited by 0
  9. 2026UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasoundcited by 0
  10. 2026A Concept-informed Interpretable Artificial Intelligence Framework for Breast Ultrasound Diagnosiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound
  2. 2026Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports
  3. 2025UMind-VL: A Generalist Ultrasound Vision-Language Model for Unified Grounded Perception and Comprehensive Interpretation
  4. 2025EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence
  5. 2025FETAL-GAUGE: A Benchmark for Assessing Vision-Language Models in Fetal Ultrasound
  6. 2025Dolphin v1.0 Technical Report
  7. 2025Adapting Vision-Language Foundation Model for Next Generation Medical Ultrasound Image Analysis
  8. 2026Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation