# Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation

arXiv: https://arxiv.org/abs/2107.09842 · HTML: https://mlsanity.com/paper/2107.09842/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 18 of 18)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images](https://mlsanity.com/paper/1808.08114/) (2018)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation](https://mlsanity.com/paper/1907.11150/) (2019)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)
- [AnatomyNet: Deep Learning for Fast and Fully Automated Whole-volume Segmentation of Head and Neck Anatomy](https://mlsanity.com/paper/1808.05238/) (2018)
- [3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes](https://mlsanity.com/paper/1711.08580/) (2017)
- [ET-Net: A Generic Edge-aTtention Guidance Network for Medical Image Segmentation](https://mlsanity.com/paper/1907.10936/) (2019)
- [Automatic Segmentation of Vestibular Schwannoma from T2-Weighted MRI by Deep Spatial Attention with Hardness-Weighted Loss](https://mlsanity.com/paper/1906.03906/) (2019)
- [AMC: Attention guided Multi-modal Correlation Learning for Image Search](https://mlsanity.com/paper/1704.00763/) (2017)
- [Hyper-Pairing Network for Multi-Phase Pancreatic Ductal Adenocarcinoma Segmentation](https://mlsanity.com/paper/1909.00906/) (2019)
- [OctopusNet: A Deep Learning Segmentation Network for Multi-modal Medical Images](https://mlsanity.com/paper/1906.02031/) (2019)
- [Simultaneous synthesis of FLAIR and segmentation of white matter hypointensities from T1 MRIs](https://mlsanity.com/paper/1808.06519/) (2018)
- [OralCam: Enabling Self-Examination and Awareness of Oral Health Using a Smartphone Camera](https://mlsanity.com/paper/2001.05621/) (2020)

## Built on by (most cited first, 30 of 47)

- [mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation](https://mlsanity.com/paper/2206.02425/) (2022)
- [UniverSeg: Universal Medical Image Segmentation](https://mlsanity.com/paper/2304.06131/) (2023)
- [Multimodal Representation Learning by Alternating Unimodal Adaptation](https://mlsanity.com/paper/2311.10707/) (2023)
- [Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality](https://mlsanity.com/paper/2310.01035/) (2023)
- [NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation](https://mlsanity.com/paper/2208.14876/) (2022)
- [Rethinking Abdominal Organ Segmentation (RAOS) in the clinical scenario: A robustness evaluation benchmark with challenging cases](https://mlsanity.com/paper/2406.13674/) (2024)
- [Medical Image Segmentation on MRI Images with Missing Modalities: A Review](https://mlsanity.com/paper/2203.06217/) (2022)
- [SegReg: Segmenting OARs by Registering MR Images and CT Annotations](https://mlsanity.com/paper/2311.06956/) (2023)
- [CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans](https://mlsanity.com/paper/2301.12291/) (2023)
- [Probabilistic Integration of Object Level Annotations in Chest X-ray Classification](https://mlsanity.com/paper/2210.06980/) (2022)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation](https://mlsanity.com/paper/2407.05540/) (2024)
- [MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Model](https://mlsanity.com/paper/2410.05905/) (2024)
- [MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training](https://mlsanity.com/paper/2411.15576/) (2024)
- [Liver Tumor Screening and Diagnosis in CT with Pixel-Lesion-Patient Network](https://mlsanity.com/paper/2307.08268/) (2023)
- [Towards the Causal Complete Cause of Multi-Modal Representation Learning](https://mlsanity.com/paper/2407.14058/) (2024)
- [SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation](https://mlsanity.com/paper/2412.12660/) (2024)
- [Completed Feature Disentanglement Learning for Multimodal MRIs Analysis](https://mlsanity.com/paper/2407.04916/) (2024)
- [Trustworthy Multi-phase Liver Tumor Segmentation via Evidence-based Uncertainty](https://mlsanity.com/paper/2305.05344/) (2023)
- [Multiscale Progressive Text Prompt Network for Medical Image Segmentation](https://mlsanity.com/paper/2307.00174/) (2023)
- [Multi-modal Learning with Missing Modality in Predicting Axillary Lymph Node Metastasis](https://mlsanity.com/paper/2401.01553/) (2024)
- [DiffSeg: A Segmentation Model for Skin Lesions Based on Diffusion Difference](https://mlsanity.com/paper/2404.16474/) (2024)
- [Lesion-Aware Cross-Phase Attention Network for Renal Tumor Subtype Classification on Multi-Phase CT Scans](https://mlsanity.com/paper/2406.16322/) (2024)
- [Joint Prediction of Meningioma Grade and Brain Invasion via Task-Aware Contrastive Learning](https://mlsanity.com/paper/2209.01517/) (2022)
- [Learning Disentangled Representations for Generalized Multi-view Clustering](https://mlsanity.com/paper/2605.15640/) (2026)
- [Opportunities and challenges in the application of large artificial intelligence models in radiology](https://mlsanity.com/paper/2403.16112/) (2024)
- [A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities](https://mlsanity.com/paper/2404.14019/) (2024)
- [Enhancing Incomplete Multi-modal Brain Tumor Segmentation with Intra-modal Asymmetry and Inter-modal Dependency](https://mlsanity.com/paper/2406.10175/) (2024)
- [Diff4MMLiTS: Advanced Multimodal Liver Tumor Segmentation via Diffusion-Based Image Synthesis and Alignment](https://mlsanity.com/paper/2412.20418/) (2024)
- [Enhancing Synthetic CT from CBCT via Multimodal Fusion: A Study on the Impact of CBCT Quality and Alignment](https://mlsanity.com/paper/2506.08716/) (2025)

## Abstract (arXiv)

Liver cancer is one of the most common cancers worldwide. Due to inconspicuous texture changes of liver tumor, contrast-enhanced computed tomography (CT) imaging is effective for the diagnosis of liver cancer. In this paper, we focus on improving automated liver tumor segmentation by integrating multi-modal CT images. To this end, we propose a novel mutual learning (ML) strategy for effective and robust multi-modal liver tumor segmentation. Different from existing multi-modal methods that fuse information from different modalities by a single model, with ML, an ensemble of modality-specific models learn collaboratively and teach each other to distill both the characteristics and the commonality between high-level representations of different modalities. The proposed ML not only enables the superiority for multi-modal learning but can also handle missing modalities by transferring knowledge from existing modalities to missing ones. Additionally, we present a modality-aware (MA) module, where the modality-specific models are interconnected and calibrated with attention weights for adaptive information exchange. The proposed modality-aware mutual learning (MAML) method achieves promising results for liver tumor segmentation on a large-scale clinical dataset. Moreover, we show the efficacy and robustness of MAML for handling missing modalities on both the liver tumor and public brain tumor (BRATS 2018) datasets. Our code is available at https://github.com/YaoZhang93/MAML.