# On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra-observer Variability in 2D Echocardiography Quality Assessment

arXiv: https://arxiv.org/abs/1911.00674 · HTML: https://mlsanity.com/paper/1911.00674/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2019.

## Built on by (most cited first, 12 of 12)

- [A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods](https://mlsanity.com/paper/2310.06873/) (2023)
- [ProtoASNet: Dynamic Prototypes for Inherently Interpretable and Uncertainty-Aware Aortic Stenosis Classification in Echocardiography](https://mlsanity.com/paper/2307.14433/) (2023)
- [Image quality assessment for machine learning tasks using meta-reinforcement learning](https://mlsanity.com/paper/2203.14258/) (2022)
- [Image quality assessment by overlapping task-specific and task-agnostic measures: application to prostate multiparametric MR images for cancer segmentation](https://mlsanity.com/paper/2202.09798/) (2022)
- [Adaptable image quality assessment using meta-reinforcement learning of task amenability](https://mlsanity.com/paper/2108.04359/) (2021)
- [PRECISE-AS: Personalized Reinforcement Learning for Efficient Point-of-Care Echocardiography in Aortic Stenosis Diagnosis](https://mlsanity.com/paper/2509.02898/) (2025)
- [Label uncertainty-guided multi-stream model for disease screening](https://mlsanity.com/paper/2201.12089/) (2022)
- [Automated Assessment of Transthoracic Echocardiogram Image Quality Using Deep Neural Networks](https://mlsanity.com/paper/2209.00976/) (2022)
- [Echocardiographic Image Quality Assessment Using Deep Neural Networks](https://mlsanity.com/paper/2209.00959/) (2022)
- [Using ScrutinAI for Visual Inspection of DNN Performance in a Medical Use Case](https://mlsanity.com/paper/2308.01220/) (2023)
- [Point Tracking as a Temporal Cue for Robust Myocardial Segmentation in Echocardiography Videos](https://mlsanity.com/paper/2601.09207/) (2026)
- [Optimizing Point-of-Care Ultrasound Video Acquisition for Probabilistic Multi-Task Heart Failure Detection](https://mlsanity.com/paper/2602.13658/) (2026)

## Abstract (arXiv)

Uncertainty of labels in clinical data resulting from intra-observer variability can have direct impact on the reliability of assessments made by deep neural networks. In this paper, we propose a method for modelling such uncertainty in the context of 2D echocardiography (echo), which is a routine procedure for detecting cardiovascular disease at point-of-care. Echo imaging quality and acquisition time is highly dependent on the operator's experience level. Recent developments have shown the possibility of automating echo image quality quantification by mapping an expert's assessment of quality to the echo image via deep learning techniques. Nevertheless, the observer variability in the expert's assessment can impact the quality quantification accuracy. Here, we aim to model the intra-observer variability in echo quality assessment as an aleatoric uncertainty modelling regression problem with the introduction of a novel method that handles the regression problem with categorical labels. A key feature of our design is that only a single forward pass is sufficient to estimate the level of uncertainty for the network output. Compared to the 0.11 ± 0.09 absolute error (in a scale from 0 to 1) archived by the conventional regression method, the proposed method brings the error down to 0.09 ± 0.08, where the improvement is statistically significant and equivalents to 5.7% test accuracy improvement. The simplicity of the proposed approach means that it could be generalized to other applications of deep learning in medical imaging, where there is often uncertainty in clinical labels.