# Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation

arXiv: https://arxiv.org/abs/2107.03442 · HTML: https://mlsanity.com/paper/2107.03442/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 10 of 10)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Gaussian Process Prior Variational Autoencoders](https://mlsanity.com/paper/1810.11738/) (2018)
- [Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation](https://mlsanity.com/paper/1907.11150/) (2019)
- [High Tissue Contrast MRI Synthesis Using Multi-Stage Attention-GAN for Glioma Segmentation](https://mlsanity.com/paper/2006.05030/) (2020)
- [PIMMS: Permutation Invariant Multi-Modal Segmentation](https://mlsanity.com/paper/1807.06537/) (2018)
- [Convolutional 3D to 2D Patch Conversion for Pixel-wise Glioma Segmentation in MRI Scans](https://mlsanity.com/paper/2010.10612/) (2020)

## Built on by (most cited first, 16 of 16)

- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Medical Image Segmentation on MRI Images with Missing Modalities: A Review](https://mlsanity.com/paper/2203.06217/) (2022)
- [Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects](https://mlsanity.com/paper/2311.02332/) (2023)
- [Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentation](https://mlsanity.com/paper/2302.04308/) (2023)
- [Distilled Prompt Learning for Incomplete Multimodal Survival Prediction](https://mlsanity.com/paper/2503.01653/) (2025)
- [Probabilistic Integration of Object Level Annotations in Chest X-ray Classification](https://mlsanity.com/paper/2210.06980/) (2022)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [Meta-Learned Modality-Weighted Knowledge Distillation for Robust Multi-Modal Learning with Missing Data](https://mlsanity.com/paper/2405.07155/) (2024)
- [Towards Optimal Patch Size in Vision Transformers for Tumor Segmentation](https://mlsanity.com/paper/2308.16598/) (2023)
- [Attention-based CT Scan Interpolation for Lesion Segmentation of Colorectal Liver Metastases](https://mlsanity.com/paper/2308.15932/) (2023)
- [Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion](https://mlsanity.com/paper/2203.04586/) (2022)
- [Finding Reproducible and Prognostic Radiomic Features in Variable Slice Thickness Contrast Enhanced CT of Colorectal Liver Metastases](https://mlsanity.com/paper/2501.11221/) (2025)
- [CCSD: Cross-Modal Compositional Self-Distillation for Robust Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2511.14599/) (2025)
- [REMIND: Rethinking Medical High-Modality Learning under Missingness--A Long-Tailed Distribution Perspective](https://mlsanity.com/paper/2603.00046/) (2026)
- [Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learning](https://mlsanity.com/paper/2406.08634/) (2024)
- [No Modality Left Behind: Dynamic Model Generation for Incomplete Medical Data](https://mlsanity.com/paper/2509.11406/) (2025)

## Abstract (arXiv)

In large studies involving multi protocol Magnetic Resonance Imaging (MRI), it can occur to miss one or more sub-modalities for a given patient owing to poor quality (e.g. imaging artifacts), failed acquisitions, or hallway interrupted imaging examinations. In some cases, certain protocols are unavailable due to limited scan time or to retrospectively harmonise the imaging protocols of two independent studies. Missing image modalities pose a challenge to segmentation frameworks as complementary information contributed by the missing scans is then lost. In this paper, we propose a novel model, Multi-modal Gaussian Process Prior Variational Autoencoder (MGP-VAE), to impute one or more missing sub-modalities for a patient scan. MGP-VAE can leverage the Gaussian Process (GP) prior on the Variational Autoencoder (VAE) to utilize the subjects/patients and sub-modalities correlations. Instead of designing one network for each possible subset of present sub-modalities or using frameworks to mix feature maps, missing data can be generated from a single model based on all the available samples. We show the applicability of MGP-VAE on brain tumor segmentation where either, two, or three of four sub-modalities may be missing. Our experiments against competitive segmentation baselines with missing sub-modality on BraTS'19 dataset indicate the effectiveness of the MGP-VAE model for segmentation tasks.