# Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities

arXiv: https://arxiv.org/abs/2104.06231 · HTML: https://mlsanity.com/paper/2104.06231/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [Pyramid Attention Network for Semantic Segmentation](https://mlsanity.com/paper/1805.10180/) (2018)
- [Brain Tumor Segmentation with Deep Neural Networks](https://mlsanity.com/paper/1505.03540/) (2015)
- [Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks](https://mlsanity.com/paper/1803.02579/) (2018)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [A review: Deep learning for medical image segmentation using multi-modality fusion](https://mlsanity.com/paper/2004.10664/) (2020)
- ['Squeeze & Excite' Guided Few-Shot Segmentation of Volumetric Images](https://mlsanity.com/paper/1902.01314/) (2019)
- [Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation](https://mlsanity.com/paper/1907.11150/) (2019)
- [Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion](https://mlsanity.com/paper/2002.09708/) (2020)
- [Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation](https://mlsanity.com/paper/1711.01468/) (2017)
- [Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge](https://mlsanity.com/paper/1802.10508/) (2018)
- [Brain Tumor Segmentation on MRI with Missing Modalities](https://mlsanity.com/paper/1904.07290/) (2019)
- [Automatic Brain Tumor Segmentation using Cascaded Anisotropic Convolutional Neural Networks](https://mlsanity.com/paper/1709.00382/) (2017)
- [One-pass Multi-task Networks with Cross-task Guided Attention for Brain Tumor Segmentation](https://mlsanity.com/paper/1906.01796/) (2019)
- [A unified representation network for segmentation with missing modalities](https://mlsanity.com/paper/1908.06683/) (2019)
- [Brain Tumor Segmentation and Survival Prediction](https://mlsanity.com/paper/1909.09399/) (2019)
- [3D Convolutional Neural Networks for Tumor Segmentation using Long-range 2D Context](https://mlsanity.com/paper/1807.08599/) (2018)
- [Brain Tumor Synthetic Segmentation in 3D Multimodal MRI Scans](https://mlsanity.com/paper/1909.13640/) (2019)
- [Multi-Resolution 3D CNN for MRI Brain Tumor Segmentation and Survival Prediction](https://mlsanity.com/paper/1911.08388/) (2019)

## Built on by (most cited first, 30 of 42)

- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2303.05302/) (2023)
- [Unified Multi-Modal Image Synthesis for Missing Modality Imputation](https://mlsanity.com/paper/2304.05340/) (2023)
- [SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities](https://mlsanity.com/paper/2204.02961/) (2022)
- [Prototype Knowledge Distillation for Medical Segmentation with Missing Modality](https://mlsanity.com/paper/2303.09830/) (2023)
- [Medical Image Segmentation on MRI Images with Missing Modalities: A Review](https://mlsanity.com/paper/2203.06217/) (2022)
- [SFusion: Self-attention based N-to-One Multimodal Fusion Block](https://mlsanity.com/paper/2208.12776/) (2022)
- [Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentation](https://mlsanity.com/paper/2302.04308/) (2023)
- [Feature-enhanced Generation and Multi-modality Fusion based Deep Neural Network for Brain Tumor Segmentation with Missing MR Modalities](https://mlsanity.com/paper/2111.04735/) (2021)
- [Gradient-Guided Modality Decoupling for Missing-Modality Robustness](https://mlsanity.com/paper/2402.16318/) (2024)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [Federated Modality-specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation](https://mlsanity.com/paper/2403.11803/) (2024)
- [Hypergraph Tversky-Aware Domain Incremental Learning for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2505.16809/) (2025)
- [3D MRI Synthesis with Slice-Based Latent Diffusion Models: Improving Tumor Segmentation Tasks in Data-Scarce Regimes](https://mlsanity.com/paper/2406.05421/) (2024)
- [MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training](https://mlsanity.com/paper/2411.15576/) (2024)
- [Meta-Learned Modality-Weighted Knowledge Distillation for Robust Multi-Modal Learning with Missing Data](https://mlsanity.com/paper/2405.07155/) (2024)
- [Feasibility of Federated Learning from Client Databases with Different Brain Diseases and MRI Modalities](https://mlsanity.com/paper/2406.11636/) (2024)
- [Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights](https://mlsanity.com/paper/2403.19882/) (2024)
- [ISA-Net: Improved spatial attention network for PET-CT tumor segmentation](https://mlsanity.com/paper/2211.02256/) (2022)
- [Bone mineral density estimation from a plain X-ray image by learning decomposition into projections of bone-segmented computed tomography](https://mlsanity.com/paper/2307.11513/) (2023)
- [Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentation](https://mlsanity.com/paper/2603.04887/) (2026)
- [Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion](https://mlsanity.com/paper/2203.04586/) (2022)
- [A Quantitative Comparison between Shannon and Tsallis Havrda Charvat Entropies Applied to Cancer Outcome Prediction](https://mlsanity.com/paper/2203.11943/) (2022)
- [Anatomical Consistency Distillation and Inconsistency Synthesis for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2408.13733/) (2024)
- [DS3-Net: Difficulty-perceived Common-to-T1ce Semi-Supervised Multimodal MRI Synthesis Network](https://mlsanity.com/paper/2203.06920/) (2022)
- [Adaptive PromptNet For Auxiliary Glioma Diagnosis without Contrast-Enhanced MRI](https://mlsanity.com/paper/2211.07966/) (2022)
- [TKN: Transformer-based Keypoint Prediction Network For Real-time Video Prediction](https://mlsanity.com/paper/2303.09807/) (2023)
- [A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities](https://mlsanity.com/paper/2404.14019/) (2024)
- [A Performance-Consistent and Computation-Efficient CNN System for High-Quality Automated Brain Tumor Segmentation](https://mlsanity.com/paper/2205.01239/) (2022)
- [Learning Multi-Modal Brain Tumor Segmentation from Privileged Semi-Paired MRI Images with Curriculum Disentanglement Learning](https://mlsanity.com/paper/2208.12781/) (2022)

## Abstract (arXiv)

Magnetic Resonance Imaging (MRI) is a widely used imaging technique to assess brain tumor. Accurately segmenting brain tumor from MR images is the key to clinical diagnostics and treatment planning. In addition, multi-modal MR images can provide complementary information for accurate brain tumor segmentation. However, it's common to miss some imaging modalities in clinical practice. In this paper, we present a novel brain tumor segmentation algorithm with missing modalities. Since it exists a strong correlation between multi-modalities, a correlation model is proposed to specially represent the latent multi-source correlation. Thanks to the obtained correlation representation, the segmentation becomes more robust in the case of missing modality. First, the individual representation produced by each encoder is used to estimate the modality independent parameter. Then, the correlation model transforms all the individual representations to the latent multi-source correlation representations. Finally, the correlation representations across modalities are fused via attention mechanism into a shared representation to emphasize the most important features for segmentation. We evaluate our model on BraTS 2018 and BraTS 2019 dataset, it outperforms the current state-of-the-art methods and produces robust results when one or more modalities are missing.