# Representation Disentanglement for Multi-modal brain MR Analysis arXiv: https://arxiv.org/abs/2102.11456 ยท HTML: https://mlsanity.com/paper/2102.11456/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Builds on 12 of its 19 references that are in the map. - First posted on arXiv 23 Feb 2021. ## Builds on (most cited first, 12 of 12) - [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019) - [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018) - [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018) - [CondConv: Conditionally Parameterized Convolutions for Efficient Inference](https://mlsanity.com/paper/1904.04971/) (2019) - [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019) - [Unsupervised Learning of Disentangled Representations from Video](https://mlsanity.com/paper/1705.10915/) (2017) - [Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations](https://mlsanity.com/paper/1903.09331/) (2019) - [CollaGAN : Collaborative GAN for Missing Image Data Imputation](https://mlsanity.com/paper/1901.09764/) (2019) - [Multi-Domain Image Completion for Random Missing Input Data](https://mlsanity.com/paper/2007.05534/) (2020) - [Domain Intersection and Domain Difference](https://mlsanity.com/paper/1908.11628/) (2019) - [Confounder-Aware Visualization of ConvNets](https://mlsanity.com/paper/1907.12727/) (2019) - [Disentangle, align and fuse for multimodal and semi-supervised image segmentation](https://mlsanity.com/paper/1911.04417/) (2019) ## Built on by (most cited first, 22 of 22) - [Learning Disentangled Representations in the Imaging Domain](https://mlsanity.com/paper/2108.12043/) (2021) - [One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation](https://mlsanity.com/paper/2204.13738/) (2022) - [Generating Realistic Brain MRIs via a Conditional Diffusion Probabilistic Model](https://mlsanity.com/paper/2212.08034/) (2022) - [Multi-Modality Cardiac Image Computing: A Survey](https://mlsanity.com/paper/2208.12881/) (2022) - [Medical Image Segmentation on MRI Images with Missing Modalities: A Review](https://mlsanity.com/paper/2203.06217/) (2022) - [HACA3: A Unified Approach for Multi-site MR Image Harmonization](https://mlsanity.com/paper/2212.06065/) (2022) - [SFusion: Self-attention based N-to-One Multimodal Fusion Block](https://mlsanity.com/paper/2208.12776/) (2022) - [Disentangling A Single MR Modality](https://mlsanity.com/paper/2205.04982/) (2022) - [Review of Disentanglement Approaches for Medical Applications -- Towards Solving the Gordian Knot of Generative Models in Healthcare](https://mlsanity.com/paper/2203.11132/) (2022) - [Cross-Modal Vertical Federated Learning for MRI Reconstruction](https://mlsanity.com/paper/2306.02673/) (2023) - [DC-Seg: Disentangled Contrastive Learning for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2505.11921/) (2025) - [MR-CLIP: Efficient Metadata-Guided Learning of MRI Contrast Representations](https://mlsanity.com/paper/2507.00043/) (2025) - [MAP: Domain Generalization via Meta-Learning on Anatomy-Consistent Pseudo-Modalities](https://mlsanity.com/paper/2309.01286/) (2023) - [Multimodal Masked Autoencoder Pre-training for 3D MRI-Based Brain Tumor Analysis with Missing Modalities](https://mlsanity.com/paper/2505.00568/) (2025) - [DIST-CLIP: Arbitrary Metadata and Image Guided MRI Harmonization via Disentangled Anatomy-Contrast Representations](https://mlsanity.com/paper/2512.07674/) (2025) - [Bayesian Unsupervised Disentanglement of Anatomy and Geometry for Deep Groupwise Image Registration](https://mlsanity.com/paper/2401.02141/) (2024) - [OWT: A Foundational Organ-Wise Tokenization Framework for Medical Imaging](https://mlsanity.com/paper/2505.04899/) (2025) - [An Adaptive, Disentangled Representation for Multidimensional MRI Reconstruction](https://mlsanity.com/paper/2512.24674/) (2025) - [Learning Multi-Modal Brain Tumor Segmentation from Privileged Semi-Paired MRI Images with Curriculum Disentanglement Learning](https://mlsanity.com/paper/2208.12781/) (2022) - [VesselMorph: Domain-Generalized Retinal Vessel Segmentation via Shape-Aware Representation](https://mlsanity.com/paper/2307.00240/) (2023) - [Enhancing Cross-Modal Medical Image Segmentation through Compositionality](https://mlsanity.com/paper/2408.11733/) (2024) - [Disentanglement and Assessment of Shortcuts in Ophthalmological Retinal Imaging Exams](https://mlsanity.com/paper/2507.09640/) (2025) ## Abstract (arXiv) Multi-modal MRIs are widely used in neuroimaging applications since different MR sequences provide complementary information about brain structures. Recent works have suggested that multi-modal deep learning analysis can benefit from explicitly disentangling anatomical (shape) and modality (appearance) information into separate image presentations. In this work, we challenge mainstream strategies by showing that they do not naturally lead to representation disentanglement both in theory and in practice. To address this issue, we propose a margin loss that regularizes the similarity in relationships of the representations across subjects and modalities. To enable robust training, we further use a conditional convolution to design a single model for encoding images of all modalities. Lastly, we propose a fusion function to combine the disentangled anatomical representations as a set of modality-invariant features for downstream tasks. We evaluate the proposed method on three multi-modal neuroimaging datasets. Experiments show that our proposed method can achieve superior disentangled representations compared to existing disentanglement strategies. Results also indicate that the fused anatomical representation has potential in the downstream task of zero-dose PET reconstruction and brain tumor segmentation. The code is available at urlhttps://github.com/ouyangjiahong/representation-disentanglement.