# A Clinician-Centered Pipeline for Annotation and Evaluation in Ultrasound AI Studies

arXiv: https://arxiv.org/abs/2606.19174 · HTML: https://mlsanity.com/paper/2606.19174/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 34 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images](https://mlsanity.com/paper/2203.12362/) (2022)
- [A Review on Deep-Learning Algorithms for Fetal Ultrasound-Image Analysis](https://mlsanity.com/paper/2201.12260/) (2022)
- [AutoFB: Automating Fetal Biometry Estimation from Standard Ultrasound Planes](https://mlsanity.com/paper/2107.05255/) (2021)
- [FetalNet: Multi-task Deep Learning Framework for Fetal Ultrasound Biometric Measurements](https://mlsanity.com/paper/2107.06943/) (2021)
- [DeepEdit: Deep Editable Learning for Interactive Segmentation of 3D Medical Images](https://mlsanity.com/paper/2305.10655/) (2023)
- [MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video](https://mlsanity.com/paper/2408.03761/) (2024)
- [Leveraging Information Divergence for Robust Semi-Supervised Fetal Ultrasound Image Segmentation](https://mlsanity.com/paper/2509.06495/) (2025)

## Abstract (arXiv)

Clinician-centered evaluation is critical for validating medical AI systems, especially in ultrasound imaging where quantitative metrics do not always capture clinical usability. Existing medical image platforms primarily focus on dataset labeling. They lack integrated support for blinded model comparison and reproducible evaluation workflows. We present a clinician-centered pipeline for remote annotation and evaluation in ultrasound AI studies. The proposed pipeline uses a centralized server and lightweight browser interfaces to enable clinicians to perform annotation, blinded ranking, and review without local dataset downloads. The pipeline also supports multi-rater participation, centralized result aggregation, and automated statistical analysis. We validate the pipeline in a fetal ultrasound segmentation study with six raters spanning expert, generalist, and non-expert experience levels. The system automatically generated Spearman correlation, Kendall's τ, and top-1 selection statistics. Results indicated moderate to strong agreement across experts and other groups. The blinded evaluation results showed a tendency for later active learning models to be preferred. These outcomes suggest that the pipeline can support clinician-centered annotation and reproducible human-acAI evaluation studies in ultrasound imaging. The proposed pipeline is available on hrefhttps://github.com/13204942/SonoRateGitHub.