# Modeling Annotation Uncertainty with Gaussian Heatmaps in Landmark Localization

arXiv: https://arxiv.org/abs/2109.09533 · HTML: https://mlsanity.com/paper/2109.09533/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 52 references that are in the map.
- First posted on arXiv 20 Sep 2021.

## Builds on (most cited first, 18 of 18)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018)
- [Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation](https://mlsanity.com/paper/1406.2984/) (2014)
- [Integrating Spatial Configuration into Heatmap Regression Based CNNs for Landmark Localization](https://mlsanity.com/paper/1908.00748/) (2019)
- [A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference](https://mlsanity.com/paper/1901.02731/) (2019)
- [On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation](https://mlsanity.com/paper/1806.02562/) (2018)
- [Cephalometric Landmark Detection by AttentiveFeature Pyramid Fusion and Regression-Voting](https://mlsanity.com/paper/1908.08841/) (2019)
- [An Attention-Guided Deep Regression Model for Landmark Detection in Cephalograms](https://mlsanity.com/paper/1906.07549/) (2019)
- [X-ray-transform Invariant Anatomical Landmark Detection for Pelvic Trauma Surgery](https://mlsanity.com/paper/1803.08608/) (2018)
- [Inherent Brain Segmentation Quality Control from Fully ConvNet Monte Carlo Sampling](https://mlsanity.com/paper/1804.07046/) (2018)
- [Semantic Alignment: Finding Semantically Consistent Ground-truth for Facial Landmark Detection](https://mlsanity.com/paper/1903.10661/) (2019)
- [Automatic Vertebra Labeling in Large-Scale 3D CT using Deep Image-to-Image Network with Message Passing and Sparsity Regularization](https://mlsanity.com/paper/1705.05998/) (2017)

## Built on by (most cited first, 6 of 6)

- [Uncertainty Estimation for Heatmap-based Landmark Localization](https://mlsanity.com/paper/2203.02351/) (2022)
- [landmarker: a Toolkit for Anatomical Landmark Localization in 2D/3D Images](https://mlsanity.com/paper/2501.10098/) (2025)
- [Reliable uncertainty quantification for 2D/3D anatomical landmark localization using multi-output conformal prediction](https://mlsanity.com/paper/2503.14106/) (2025)
- [Heatmap Regression for Lesion Detection using Pointwise Annotations](https://mlsanity.com/paper/2208.05939/) (2022)
- [Reliability of deep learning models for anatomical landmark detection: The role of inter-rater variability](https://mlsanity.com/paper/2411.17850/) (2024)
- [Uncertainty Propagation for Echocardiography Clinical Metric Estimation via Contour Sampling](https://mlsanity.com/paper/2502.12713/) (2025)

## Abstract (arXiv)

In landmark localization, due to ambiguities in defining their exact position, landmark annotations may suffer from large observer variabilities, which result in uncertain annotations. To model the annotation ambiguities of the training dataset, we propose to learn anisotropic Gaussian parameters modeling the shape of the target heatmap during optimization. Furthermore, our method models the prediction uncertainty of individual samples by fitting anisotropic Gaussian functions to the predicted heatmaps during inference. Besides state-of-the-art results, our experiments on datasets of hand radiographs and lateral cephalograms also show that Gaussian functions are correlated with both localization accuracy and observer variability. As a final experiment, we show the importance of integrating the uncertainty into decision making by measuring the influence of the predicted location uncertainty on the classification of anatomical abnormalities in lateral cephalograms.