# Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion

arXiv: https://arxiv.org/abs/2203.04586 · HTML: https://mlsanity.com/paper/2203.04586/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 17 of 17)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network](https://mlsanity.com/paper/1802.09655/) (2018)
- [Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion](https://mlsanity.com/paper/2002.09708/) (2020)
- [HyperDense-Net: A hyper-densely connected CNN for multi-modal image segmentation](https://mlsanity.com/paper/1804.02967/) (2018)
- [Unsupervised Domain Adaptation via Disentangled Representations: Application to Cross-Modality Liver Segmentation](https://mlsanity.com/paper/1907.13590/) (2019)
- [Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities](https://mlsanity.com/paper/2104.06231/) (2021)
- [Brain Tumor Segmentation using an Ensemble of 3D U-Nets and Overall Survival Prediction using Radiomic Features](https://mlsanity.com/paper/1812.01049/) (2018)
- [MRI Tumor Segmentation with Densely Connected 3D CNN](https://mlsanity.com/paper/1802.02427/) (2018)
- [Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation](https://mlsanity.com/paper/2107.03442/) (2021)
- [Multi-Modality Generative Adversarial Networks with Tumor Consistency Loss for Brain MR Image Synthesis](https://mlsanity.com/paper/2005.00925/) (2020)

## Built on by (most cited first, 2 of 2)

- [MouseGAN++: Unsupervised Disentanglement and Contrastive Representation for Multiple MRI Modalities Synthesis and Structural Segmentation of Mouse Brain](https://mlsanity.com/paper/2212.01825/) (2022)
- [DS3-Net: Difficulty-perceived Common-to-T1ce Semi-Supervised Multimodal MRI Synthesis Network](https://mlsanity.com/paper/2203.06920/) (2022)

## Abstract (arXiv)

Multi-modal magnetic resonance (MR) imaging provides great potential for diagnosing and analyzing brain gliomas. In clinical scenarios, common MR sequences such as T1, T2 and FLAIR can be obtained simultaneously in a single scanning process. However, acquiring contrast enhanced modalities such as T1ce requires additional time, cost, and injection of contrast agent. As such, it is clinically meaningful to develop a method to synthesize unavailable modalities which can also be used as additional inputs to downstream tasks (e.g., brain tumor segmentation) for performance enhancing. In this work, we propose an end-to-end framework named Modality-Level Attention Fusion Network (MAF-Net), wherein we innovatively conduct patchwise contrastive learning for extracting multi-modal latent features and dynamically assigning attention weights to fuse different modalities. Through extensive experiments on BraTS2020, our proposed MAF-Net is found to yield superior T1ce synthesis performance (SSIM of 0.8879 and PSNR of 22.78) and accurate brain tumor segmentation (mean Dice scores of 67.9%, 41.8% and 88.0% on segmenting the tumor core, enhancing tumor and whole tumor).