# Joint Modeling of Image and Label Statistics for Enhancing Model Generalizability of Medical Image Segmentation

arXiv: https://arxiv.org/abs/2206.04336 · HTML: https://mlsanity.com/paper/2206.04336/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [Data augmentation using learned transformations for one-shot medical image segmentation](https://mlsanity.com/paper/1902.09383/) (2019)
- [Multivariate mixture model for myocardium segmentation combining multi-source images](https://mlsanity.com/paper/1612.08820/) (2016)
- [Causality-inspired Single-source Domain Generalization for Medical Image Segmentation](https://mlsanity.com/paper/2111.12525/) (2021)
- [Overfitting of neural nets under class imbalance: Analysis and improvements for segmentation](https://mlsanity.com/paper/1907.10982/) (2019)

## Built on by (most cited first, 5 of 5)

- [BayeSeg: Bayesian Modeling for Medical Image Segmentation with Interpretable Generalizability](https://mlsanity.com/paper/2303.01710/) (2023)
- [Evidential Concept Embedding Models: Towards Reliable Concept Explanations for Skin Disease Diagnosis](https://mlsanity.com/paper/2406.19130/) (2024)
- [KA2ER: Knowledge Adaptive Amalgamation of ExpeRts for Medical Images Segmentation](https://mlsanity.com/paper/2410.21085/) (2024)
- [Style Content Decomposition-based Data Augmentation for Domain Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2502.20619/) (2025)
- [Reverse Imaging for Wide-spectrum Generalization of Cardiac MRI Segmentation](https://mlsanity.com/paper/2508.21254/) (2025)

## Abstract (arXiv)

Although supervised deep-learning has achieved promising performance in medical image segmentation, many methods cannot generalize well on unseen data, limiting their real-world applicability. To address this problem, we propose a deep learning-based Bayesian framework, which jointly models image and label statistics, utilizing the domain-irrelevant contour of a medical image for segmentation. Specifically, we first decompose an image into components of contour and basis. Then, we model the expected label as a variable only related to the contour. Finally, we develop a variational Bayesian framework to infer the posterior distributions of these variables, including the contour, the basis, and the label. The framework is implemented with neural networks, thus is referred to as deep Bayesian segmentation. Results on the task of cross-sequence cardiac MRI segmentation show that our method set a new state of the art for model generalizability. Particularly, the BayeSeg model trained with LGE MRI generalized well on T2 images and outperformed other models with great margins, i.e., over 0.47 in terms of average Dice. Our code is available at https://zmiclab.github.io/projects.html.