# Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality

arXiv: https://arxiv.org/abs/2310.01035 · HTML: https://mlsanity.com/paper/2310.01035/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 15 of 15)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation](https://mlsanity.com/paper/2206.02425/) (2022)
- [Unpaired Multi-modal Segmentation via Knowledge Distillation](https://mlsanity.com/paper/2001.03111/) (2020)
- [Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation](https://mlsanity.com/paper/1907.11150/) (2019)
- [Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion](https://mlsanity.com/paper/2002.09708/) (2020)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)
- [Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation](https://mlsanity.com/paper/2107.09842/) (2021)
- [ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2106.14591/) (2021)
- [Brain Tumor Segmentation on MRI with Missing Modalities](https://mlsanity.com/paper/1904.07290/) (2019)
- [Soft Expert Reward Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2007.10835/) (2020)
- [Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering](https://mlsanity.com/paper/1704.03162/) (2017)
- [Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descent](https://mlsanity.com/paper/1607.01981/) (2016)
- [Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Prediction](https://mlsanity.com/paper/2207.10851/) (2022)

## Built on by (most cited first, 30 of 45)

- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities](https://mlsanity.com/paper/2404.16456/) (2024)
- [Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities](https://mlsanity.com/paper/2407.11351/) (2024)
- [Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning](https://mlsanity.com/paper/2411.02793/) (2024)
- [Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects](https://mlsanity.com/paper/2311.02332/) (2023)
- [What You Have is What You Track: Adaptive and Robust Multimodal Tracking](https://mlsanity.com/paper/2507.05899/) (2025)
- [Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation](https://mlsanity.com/paper/2401.16923/) (2024)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation](https://mlsanity.com/paper/2411.17141/) (2024)
- [Incomplete Multimodal Industrial Anomaly Detection via Cross-Modal Distillation](https://mlsanity.com/paper/2405.13571/) (2024)
- [ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalities](https://mlsanity.com/paper/2407.03836/) (2024)
- [Towards the Causal Complete Cause of Multi-Modal Representation Learning](https://mlsanity.com/paper/2407.14058/) (2024)
- [G2D: Boosting Multimodal Learning with Gradient-Guided Distillation](https://mlsanity.com/paper/2506.21514/) (2025)
- [Distilling Cross-Modal Knowledge via Feature Disentanglement](https://mlsanity.com/paper/2511.19887/) (2025)
- [No Modality Left Behind: Adapting to Missing Modalities via Knowledge Distillation for Brain Tumor Segmentation](https://mlsanity.com/paper/2509.15017/) (2025)
- [Learning Task-Agnostic Representations through Multi-Teacher Distillation](https://mlsanity.com/paper/2510.18680/) (2025)
- [Calibrated Multimodal Representation Learning with Missing Modalities](https://mlsanity.com/paper/2511.12034/) (2025)
- [Enhancing Single-Slice Segmentation with 3D-to-2D Unpaired Scan Distillation](https://mlsanity.com/paper/2406.12254/) (2024)
- [Human-AI Collaborative Multi-modal Multi-rater Learning for Endometriosis Diagnosis](https://mlsanity.com/paper/2409.02046/) (2024)
- [Non-target Divergence Hypothesis: Toward Understanding Domain Gaps in Cross-Modal Knowledge Distillation](https://mlsanity.com/paper/2409.02438/) (2024)
- [Robust Multimodal Learning for Ophthalmic Disease Grading via Disentangled Representation](https://mlsanity.com/paper/2503.05319/) (2025)
- [MGML: A Plug-and-Play Meta-Guided Multi-Modal Learning Framework for Incomplete Multimodal Brain Tumor Segmentation](https://mlsanity.com/paper/2512.23936/) (2025)
- [SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data](https://mlsanity.com/paper/2603.02505/) (2026)
- [Decoupling Feature Representations of Ego and Other Modalities for Incomplete Multi-modal Brain Tumor Segmentation](https://mlsanity.com/paper/2408.08708/) (2024)
- [Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalities](https://mlsanity.com/paper/2408.14585/) (2024)
- [Selecting the Best Sequential Transfer Path for Medical Image Segmentation with Limited Labeled Data](https://mlsanity.com/paper/2410.06892/) (2024)
- [Head and Neck Tumor Segmentation of MRI from Pre- and Mid-radiotherapy with Pre-training, Data Augmentation and Dual Flow UNet](https://mlsanity.com/paper/2412.14846/) (2024)
- [Multi-modal Contrastive Learning for Tumor-specific Missing Modality Synthesis](https://mlsanity.com/paper/2502.19390/) (2025)
- [PGAD: Prototype-Guided Adaptive Distillation for Multi-Modal Learning in AD Diagnosis](https://mlsanity.com/paper/2503.04836/) (2025)
- [DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation](https://mlsanity.com/paper/2506.11691/) (2025)

## Abstract (arXiv)

The problem of missing modalities is both critical and non-trivial to be handled in multi-modal models. It is common for multi-modal tasks that certain modalities contribute more compared to other modalities, and if those important modalities are missing, the model performance drops significantly. Such fact remains unexplored by current multi-modal approaches that recover the representation from missing modalities by feature reconstruction or blind feature aggregation from other modalities, instead of extracting useful information from the best performing modalities. In this paper, we propose a Learnable Cross-modal Knowledge Distillation (LCKD) model to adaptively identify important modalities and distil knowledge from them to help other modalities from the cross-modal perspective for solving the missing modality issue. Our approach introduces a teacher election procedure to select the most ``qualified'' teachers based on their single modality performance on certain tasks. Then, cross-modal knowledge distillation is performed between teacher and student modalities for each task to push the model parameters to a point that is beneficial for all tasks. Hence, even if the teacher modalities for certain tasks are missing during testing, the available student modalities can accomplish the task well enough based on the learned knowledge from their automatically elected teacher modalities. Experiments on the Brain Tumour Segmentation Dataset 2018 (BraTS2018) shows that LCKD outperforms other methods by a considerable margin, improving the state-of-the-art performance by 3.61% for enhancing tumour, 5.99% for tumour core, and 3.76% for whole tumour in terms of segmentation Dice score.