# Exploiting Partial Common Information Microstructure for Multi-Modal Brain Tumor Segmentation

arXiv: https://arxiv.org/abs/2302.02521 · HTML: https://mlsanity.com/paper/2302.02521/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 65 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation](https://mlsanity.com/paper/1809.10486/) (2018)
- [DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation](https://mlsanity.com/paper/1604.02677/) (2016)
- [U-Net Transformer: Self and Cross Attention for Medical Image Segmentation](https://mlsanity.com/paper/2103.06104/) (2021)
- [Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge](https://mlsanity.com/paper/1802.10508/) (2018)
- [Automatic Brain Tumor Segmentation using Cascaded Anisotropic Convolutional Neural Networks](https://mlsanity.com/paper/1709.00382/) (2017)
- [Survival of the strictest: Stable and unstable equilibria under regularized learning with partial information](https://mlsanity.com/paper/2101.04667/) (2021)
- [An Efficient Approach to Informative Feature Extraction from Multimodal Data](https://mlsanity.com/paper/1811.08979/) (2018)
- [An Information Theoretic Interpretation to Deep Neural Networks](https://mlsanity.com/paper/1905.06600/) (2019)
- [An Optimization-Based Meta-Learning Model for MRI Reconstruction with Diverse Dataset](https://mlsanity.com/paper/2110.00715/) (2021)
- [Modality-Pairing Learning for Brain Tumor Segmentation](https://mlsanity.com/paper/2010.09277/) (2020)
- [A Learnable Variational Model for Joint Multimodal MRI Reconstruction and Synthesis](https://mlsanity.com/paper/2204.03804/) (2022)
- [A Bayesian Optimization Framework for Finding Local Optima in Expensive Multi-Modal Functions](https://mlsanity.com/paper/2210.06635/) (2022)
- [H2NF-Net for Brain Tumor Segmentation using Multimodal MR Imaging: 2nd Place Solution to BraTS Challenge 2020 Segmentation Task](https://mlsanity.com/paper/2012.15318/) (2020)
- [Multi-Topic Multi-Document Summarizer](https://mlsanity.com/paper/1401.0640/) (2014)
- [Network Maximal Correlation](https://mlsanity.com/paper/1606.04789/) (2016)
- [An Information-theoretic Approach to Unsupervised Feature Selection for High-Dimensional Data](https://mlsanity.com/paper/1910.03196/) (2019)

## Built on by (most cited first, 3 of 3)

- [MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization](https://mlsanity.com/paper/2302.10418/) (2023)
- [Bayesian Optimization through Gaussian Cox Process Models for Spatio-temporal Data](https://mlsanity.com/paper/2401.14544/) (2024)
- [Using Diffusion Models as Generative Replay in Continual Federated Learning -- What will Happen?](https://mlsanity.com/paper/2411.06618/) (2024)

## Abstract (arXiv)

Learning with multiple modalities is crucial for automated brain tumor segmentation from magnetic resonance imaging data. Explicitly optimizing the common information shared among all modalities (e.g., by maximizing the total correlation) has been shown to achieve better feature representations and thus enhance the segmentation performance. However, existing approaches are oblivious to partial common information shared by subsets of the modalities. In this paper, we show that identifying such partial common information can significantly boost the discriminative power of image segmentation models. In particular, we introduce a novel concept of partial common information mask (PCI-mask) to provide a fine-grained characterization of what partial common information is shared by which subsets of the modalities. By solving a masked correlation maximization and simultaneously learning an optimal PCI-mask, we identify the latent microstructure of partial common information and leverage it in a self-attention module to selectively weight different feature representations in multi-modal data. We implement our proposed framework on the standard U-Net. Our experimental results on the Multi-modal Brain Tumor Segmentation Challenge (BraTS) datasets outperform those of state-of-the-art segmentation baselines, with validation Dice similarity coefficients of 0.920, 0.897, 0.837 for the whole tumor, tumor core, and enhancing tumor on BraTS-2020.