# A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities

arXiv: https://arxiv.org/abs/1905.13077 · HTML: https://mlsanity.com/paper/1905.13077/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2019.

## Built on by (most cited first, 30 of 65)

- [Diffusion Models for Implicit Image Segmentation Ensembles](https://mlsanity.com/paper/2112.03145/) (2021)
- [Ambiguous Medical Image Segmentation using Diffusion Models](https://mlsanity.com/paper/2304.04745/) (2023)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [Uncertainty Inspired Underwater Image Enhancement](https://mlsanity.com/paper/2207.09689/) (2022)
- [Stochastic Segmentation with Conditional Categorical Diffusion Models](https://mlsanity.com/paper/2303.08888/) (2023)
- [Transframer: Arbitrary Frame Prediction with Generative Models](https://mlsanity.com/paper/2203.09494/) (2022)
- [BerDiff: Conditional Bernoulli Diffusion Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.04429/) (2023)
- [FlowSDF: Flow Matching for Medical Image Segmentation Using Distance Transforms](https://mlsanity.com/paper/2405.18087/) (2024)
- [Tyche: Stochastic In-Context Learning for Medical Image Segmentation](https://mlsanity.com/paper/2401.13650/) (2024)
- [Using Soft Labels to Model Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/2109.12622/) (2021)
- [Safety-aware Motion Prediction with Unseen Vehicles for Autonomous Driving](https://mlsanity.com/paper/2109.01510/) (2021)
- [Improving Aleatoric Uncertainty Quantification in Multi-Annotated Medical Image Segmentation with Normalizing Flows](https://mlsanity.com/paper/2108.02155/) (2021)
- [Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Prediction](https://mlsanity.com/paper/2207.10851/) (2022)
- [Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts](https://mlsanity.com/paper/2212.07328/) (2022)
- [That Label's Got Style: Handling Label Style Bias for Uncertain Image Segmentation](https://mlsanity.com/paper/2303.15850/) (2023)
- [Modeling Annotator Preference and Stochastic Annotation Error for Medical Image Segmentation](https://mlsanity.com/paper/2111.13410/) (2021)
- [Generalized Probabilistic U-Net for medical image segementation](https://mlsanity.com/paper/2207.12872/) (2022)
- [Uncertainty Estimation and Propagation in Accelerated MRI Reconstruction](https://mlsanity.com/paper/2308.02631/) (2023)
- [A Multi-Resolution Framework for U-Nets with Applications to Hierarchical VAEs](https://mlsanity.com/paper/2301.08187/) (2023)
- [Laplacian Segmentation Networks Improve Epistemic Uncertainty Quantification](https://mlsanity.com/paper/2303.13123/) (2023)
- [Diversified and Personalized Multi-rater Medical Image Segmentation](https://mlsanity.com/paper/2403.13417/) (2024)
- [Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design](https://mlsanity.com/paper/2205.13927/) (2022)
- [Uncertainty-Guided Lung Nodule Segmentation with Feature-Aware Attention](https://mlsanity.com/paper/2110.12372/) (2021)
- [A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation](https://mlsanity.com/paper/2411.16370/) (2024)
- [SAMCT: Segment Any CT Allowing Labor-Free Task-Indicator Prompts](https://mlsanity.com/paper/2403.13258/) (2024)
- [Bayesian neural networks for predicting uncertainty in full-field material response](https://mlsanity.com/paper/2406.14838/) (2024)
- [MedUHIP: Towards Human-In-the-Loop Medical Segmentation](https://mlsanity.com/paper/2408.01620/) (2024)
- [Aleatoric Uncertainty Medical Image Segmentation Estimation via Flow Matching](https://mlsanity.com/paper/2507.22418/) (2025)
- [CroBIM-U: Uncertainty-Driven Referring Remote Sensing Image Segmentation](https://mlsanity.com/paper/2601.03490/) (2026)
- [Hypernet-Ensemble Learning of Segmentation Probability for Medical Image Segmentation with Ambiguous Labels](https://mlsanity.com/paper/2112.06693/) (2021)

## Abstract (arXiv)

Medical imaging only indirectly measures the molecular identity of the tissue within each voxel, which often produces only ambiguous image evidence for target measures of interest, like semantic segmentation. This diversity and the variations of plausible interpretations are often specific to given image regions and may thus manifest on various scales, spanning all the way from the pixel to the image level. In order to learn a flexible distribution that can account for multiple scales of variations, we propose the Hierarchical Probabilistic U-Net, a segmentation network with a conditional variational auto-encoder (cVAE) that uses a hierarchical latent space decomposition. We show that this model formulation enables sampling and reconstruction of segmenations with high fidelity, i.e. with finely resolved detail, while providing the flexibility to learn complex structured distributions across scales. We demonstrate these abilities on the task of segmenting ambiguous medical scans as well as on instance segmentation of neurobiological and natural images. Our model automatically separates independent factors across scales, an inductive bias that we deem beneficial in structured output prediction tasks beyond segmentation.