# BayeSeg: Bayesian Modeling for Medical Image Segmentation with Interpretable Generalizability

arXiv: https://arxiv.org/abs/2303.01710 · HTML: https://mlsanity.com/paper/2303.01710/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 61 references that are in the map.
- First posted on arXiv 3 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Generalizing to Unseen Domains via Adversarial Data Augmentation](https://mlsanity.com/paper/1805.12018/) (2018)
- [Domain Generalization via Model-Agnostic Learning of Semantic Features](https://mlsanity.com/paper/1910.13580/) (2019)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [A Survey on Neural Network Interpretability](https://mlsanity.com/paper/2012.14261/) (2020)
- [FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space](https://mlsanity.com/paper/2103.06030/) (2021)
- [Robust and Generalizable Visual Representation Learning via Random Convolutions](https://mlsanity.com/paper/2007.13003/) (2020)
- [Multivariate mixture model for myocardium segmentation combining multi-source images](https://mlsanity.com/paper/1612.08820/) (2016)
- [Uncertainty Modeling for Out-of-Distribution Generalization](https://mlsanity.com/paper/2202.03958/) (2022)
- [Shape-aware Meta-learning for Generalizing Prostate MRI Segmentation to Unseen Domains](https://mlsanity.com/paper/2007.02035/) (2020)
- [Causality-inspired Single-source Domain Generalization for Medical Image Segmentation](https://mlsanity.com/paper/2111.12525/) (2021)
- [DIVA: Domain Invariant Variational Autoencoders](https://mlsanity.com/paper/1905.10427/) (2019)
- [Causal Machine Learning: A Survey and Open Problems](https://mlsanity.com/paper/2206.15475/) (2022)
- [Domain Generalization with Domain-Specific Aggregation Modules](https://mlsanity.com/paper/1809.10966/) (2018)
- [Realistic Adversarial Data Augmentation for MR Image Segmentation](https://mlsanity.com/paper/2006.13322/) (2020)
- [Semi-supervised Meta-learning with Disentanglement for Domain-generalised Medical Image Segmentation](https://mlsanity.com/paper/2106.13292/) (2021)
- [Adversarial Consistency for Single Domain Generalization in Medical Image Segmentation](https://mlsanity.com/paper/2206.13737/) (2022)
- [Bayesian Image Super-Resolution with Deep Modeling of Image Statistics](https://mlsanity.com/paper/2204.00623/) (2022)
- [Robust White Matter Hyperintensity Segmentation on Unseen Domain](https://mlsanity.com/paper/2102.06650/) (2021)
- [Joint Modeling of Image and Label Statistics for Enhancing Model Generalizability of Medical Image Segmentation](https://mlsanity.com/paper/2206.04336/) (2022)

## Built on by (most cited first, 30 of 42)

- [Domain Generalization for Medical Image Analysis: A Review](https://mlsanity.com/paper/2310.08598/) (2023)
- [Learning General-Purpose Biomedical Volume Representations using Randomized Synthesis](https://mlsanity.com/paper/2411.02372/) (2024)
- [Dino U-Net: Exploiting High-Fidelity Dense Features from Foundation Models for Medical Image Segmentation](https://mlsanity.com/paper/2508.20909/) (2025)
- [MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output](https://mlsanity.com/paper/2510.10011/) (2025)
- [Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Survey](https://mlsanity.com/paper/2405.01636/) (2024)
- [Integrating Deep Metric Learning with Coreset for Active Learning in 3D Segmentation](https://mlsanity.com/paper/2411.15763/) (2024)
- [MedSG-Bench: A Benchmark for Medical Image Sequences Grounding](https://mlsanity.com/paper/2505.11852/) (2025)
- [Evidential Concept Embedding Models: Towards Reliable Concept Explanations for Skin Disease Diagnosis](https://mlsanity.com/paper/2406.19130/) (2024)
- [Generalizable and Explainable Deep Learning for Medical Image Computing: An Overview](https://mlsanity.com/paper/2503.08420/) (2025)
- [Domain Game: Disentangle Anatomical Feature for Single Domain Generalized Segmentation](https://mlsanity.com/paper/2406.02125/) (2024)
- [ModelMix: A New Model-Mixup Strategy to Minimize Vicinal Risk across Tasks for Few-scribble based Cardiac Segmentation](https://mlsanity.com/paper/2406.13237/) (2024)
- [CONSIGN: Conformal Segmentation Informed by Spatial Groupings via Decomposition](https://mlsanity.com/paper/2505.14113/) (2025)
- [Multi-Modal Explainable Medical AI Assistant for Trustworthy Human-AI Collaboration](https://mlsanity.com/paper/2505.06898/) (2025)
- [Augmentation-based Domain Generalization and Joint Training from Multiple Source Domains for Whole Heart Segmentation](https://mlsanity.com/paper/2508.04552/) (2025)
- [Bayesian Unsupervised Disentanglement of Anatomy and Geometry for Deep Groupwise Image Registration](https://mlsanity.com/paper/2401.02141/) (2024)
- [Bayesian Uncertainty Estimation by Hamiltonian Monte Carlo: Applications to Cardiac MRI Segmentation](https://mlsanity.com/paper/2403.02311/) (2024)
- [Synthetic Data Generation for 3D Myocardium Deformation Analysis](https://mlsanity.com/paper/2406.01040/) (2024)
- [Cross-Domain Distribution Alignment for Segmentation of Private Unannotated 3D Medical Images](https://mlsanity.com/paper/2410.09210/) (2024)
- [Preserving Cardiac Integrity: A Topology-Infused Approach to Whole Heart Segmentation](https://mlsanity.com/paper/2410.10551/) (2024)
- [KA2ER: Knowledge Adaptive Amalgamation of ExpeRts for Medical Images Segmentation](https://mlsanity.com/paper/2410.21085/) (2024)
- [Style Content Decomposition-based Data Augmentation for Domain Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2502.20619/) (2025)
- [MedCL: Learning Consistent Anatomy Distribution for Scribble-supervised Medical Image Segmentation](https://mlsanity.com/paper/2503.22890/) (2025)
- [Uncertainty-Aware Multi-Expert Knowledge Distillation for Imbalanced Disease Grading](https://mlsanity.com/paper/2505.00592/) (2025)
- [MAMBO-NET: Multi-Causal Aware Modeling Backdoor-Intervention Optimization for Medical Image Segmentation Network](https://mlsanity.com/paper/2505.21874/) (2025)
- [Reverse Imaging for Wide-spectrum Generalization of Cardiac MRI Segmentation](https://mlsanity.com/paper/2508.21254/) (2025)
- [Scalable Mean-Field Variational Inference via Preconditioned Primal-Dual Optimization](https://mlsanity.com/paper/2602.07632/) (2026)
- [LayersReg: A Layer-by-Layer Progressive Regressor for Reliable Intraoperative 3D/2D Registration](https://mlsanity.com/paper/2606.26647/) (2026)
- [Gaussian Process Emulators for Few-Shot Segmentation in Cardiac MRI](https://mlsanity.com/paper/2411.06911/) (2024)
- [L-FUSION: Laplacian Fetal Ultrasound Segmentation & Uncertainty Estimation](https://mlsanity.com/paper/2503.05245/) (2025)
- [Decentralized LoRA augmented transformer with multi-scale feature learning for secured eye diagnosis](https://mlsanity.com/paper/2505.06982/) (2025)

## Abstract (arXiv)

Due to the cross-domain distribution shift aroused from diverse medical imaging systems, many deep learning segmentation methods fail to perform well on unseen data, which limits their real-world applicability. Recent works have shown the benefits of extracting domain-invariant representations on domain generalization. However, the interpretability of domain-invariant features remains a great challenge. To address this problem, we propose an interpretable Bayesian framework (BayeSeg) through Bayesian modeling of image and label statistics to enhance model generalizability for medical image segmentation. Specifically, we first decompose an image into a spatial-correlated variable and a spatial-variant variable, assigning hierarchical Bayesian priors to explicitly force them to model the domain-stable shape and domain-specific appearance information respectively. Then, we model the segmentation as a locally smooth variable only related to the shape. Finally, we develop a variational Bayesian framework to infer the posterior distributions of these explainable variables. The framework is implemented with neural networks, and thus is referred to as deep Bayesian segmentation. Quantitative and qualitative experimental results on prostate segmentation and cardiac segmentation tasks have shown the effectiveness of our proposed method. Moreover, we investigated the interpretability of BayeSeg by explaining the posteriors and analyzed certain factors that affect the generalization ability through further ablation studies. Our code will be released via https://zmiclab.github.io/projects.html, once the manuscript is accepted for publication.