# Mirror U-Net: Marrying Multimodal Fission with Multi-task Learning for Semantic Segmentation in Medical Imaging

arXiv: https://arxiv.org/abs/2303.07126 · HTML: https://mlsanity.com/paper/2303.07126/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Attentive Single-Tasking of Multiple Tasks](https://mlsanity.com/paper/1904.08918/) (2019)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network](https://mlsanity.com/paper/1802.09655/) (2018)
- [Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge](https://mlsanity.com/paper/1802.10508/) (2018)
- [Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition](https://mlsanity.com/paper/2112.10992/) (2021)
- [Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data](https://mlsanity.com/paper/1805.11264/) (2018)
- [Multimodal Spatial Attention Module for Targeting Multimodal PET-CT Lung Tumor Segmentation](https://mlsanity.com/paper/2007.14728/) (2020)
- [Deep Learning with Mixed Supervision for Brain Tumor Segmentation](https://mlsanity.com/paper/1812.04571/) (2018)
- [DeepMTS: Deep Multi-task Learning for Survival Prediction in Patients with Advanced Nasopharyngeal Carcinoma using Pretreatment PET/CT](https://mlsanity.com/paper/2109.07711/) (2021)
- [Cross-modality (CT-MRI) prior augmented deep learning for robust lung tumor segmentation from small MR datasets](https://mlsanity.com/paper/1901.11369/) (2019)
- [Exploring Vanilla U-Net for Lesion Segmentation from Whole-body FDG-PET/CT Scans](https://mlsanity.com/paper/2210.07490/) (2022)
- [Floors are Flat: Leveraging Semantics for Real-Time Surface Normal Prediction](https://mlsanity.com/paper/1906.06792/) (2019)
- [Multimodal PET/CT Tumour Segmentation and Prediction of Progression-Free Survival using a Full-Scale UNet with Attention](https://mlsanity.com/paper/2111.03848/) (2021)
- [Multimodal Interactive Lung Lesion Segmentation: A Framework for Annotating PET/CT Images based on Physiological and Anatomical Cues](https://mlsanity.com/paper/2301.09914/) (2023)
- [AutoPET Challenge: Combining nn-Unet with Swin UNETR Augmented by Maximum Intensity Projection Classifier](https://mlsanity.com/paper/2209.01112/) (2022)
- [AutoPET Challenge 2022: Automatic Segmentation of Whole-body Tumor Lesion Based on Deep Learning and FDG PET/CT](https://mlsanity.com/paper/2209.01212/) (2022)
- [ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization](https://mlsanity.com/paper/2208.09414/) (2022)
- [AutoPET Challenge 2022: Step-by-Step Lesion Segmentation in Whole-body FDG-PET/CT](https://mlsanity.com/paper/2209.09199/) (2022)
- [PriorNet: lesion segmentation in PET-CT including prior tumor appearance information](https://mlsanity.com/paper/2210.02203/) (2022)
- [Whole-body tumor segmentation of 18F -FDG PET/CT using a cascaded and ensembled convolutional neural networks](https://mlsanity.com/paper/2210.08068/) (2022)
- [Automatic Tumor Segmentation via False Positive Reduction Network for Whole-Body Multi-Modal PET/CT Images](https://mlsanity.com/paper/2209.07705/) (2022)
- [Whole-Body Lesion Segmentation in 18F-FDG PET/CT](https://mlsanity.com/paper/2209.07851/) (2022)

## Built on by (most cited first, 14 of 14)

- [Towards Unifying Anatomy Segmentation: Automated Generation of a Full-body CT Dataset via Knowledge Aggregation and Anatomical Guidelines](https://mlsanity.com/paper/2307.13375/) (2023)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training](https://mlsanity.com/paper/2411.15576/) (2024)
- [Comprehensive framework for evaluation of deep neural networks in detection and quantification of lymphoma from PET/CT images: clinical insights, pitfalls, and observer agreement analyses](https://mlsanity.com/paper/2311.09614/) (2023)
- [Sliding Window FastEdit: A Framework for Lesion Annotation in Whole-body PET Images](https://mlsanity.com/paper/2311.14482/) (2023)
- [LF2AR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Models](https://mlsanity.com/paper/2503.06211/) (2025)
- [AutoPET Challenge 2023: Sliding Window-based Optimization of U-Net](https://mlsanity.com/paper/2309.12114/) (2023)
- [H2ASeg: Hierarchical Adaptive Interaction and Weighting Network for Tumor Segmentation in PET/CT Images](https://mlsanity.com/paper/2403.18339/) (2024)
- [Scale-aware Super-resolution Network with Dual Affinity Learning for Lesion Segmentation from Medical Images](https://mlsanity.com/paper/2305.19063/) (2023)
- [Multi-modal Evidential Fusion Network for Trustworthy PET/CT Tumor Segmentation](https://mlsanity.com/paper/2406.18327/) (2024)
- [Probabilistic Hyper-Graphs using Multiple Randomly Masked Autoencoders for Semi-supervised Multi-modal Multi-task Learning](https://mlsanity.com/paper/2510.10068/) (2025)
- [M2Diff: Multi-Modality Multi-Task Enhanced Diffusion Model for MRI-Guided Low-Dose PET Enhancement](https://mlsanity.com/paper/2603.09075/) (2026)
- [A mirror-Unet architecture for PET/CT lesion segmentation](https://mlsanity.com/paper/2309.13398/) (2023)
- [A-QCF-Net: An Adaptive Quaternion Cross-Fusion Network for Multimodal Liver Tumor Segmentation from Unpaired Datasets](https://mlsanity.com/paper/2512.21760/) (2025)

## Abstract (arXiv)

Positron Emission Tomography (PET) and Computer Tomography (CT) are routinely used together to detect tumors. PET/CT segmentation models can automate tumor delineation, however, current multimodal models do not fully exploit the complementary information in each modality, as they either concatenate PET and CT data or fuse them at the decision level. To combat this, we propose Mirror U-Net, which replaces traditional fusion methods with multimodal fission by factorizing the multimodal representation into modality-specific branches and an auxiliary multimodal decoder. At these branches, Mirror U-Net assigns a task tailored to each modality to reinforce unimodal features while preserving multimodal features in the shared representation. In contrast to previous methods that use either fission or multi-task learning, Mirror U-Net combines both paradigms in a unified framework. We explore various task combinations and examine which parameters to share in the model. We evaluate Mirror U-Net on the AutoPET PET/CT and on the multimodal MSD BrainTumor datasets, demonstrating its effectiveness in multimodal segmentation and achieving state-of-the-art performance on both datasets. Our code will be made publicly available.