# The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation

arXiv: https://arxiv.org/abs/2206.06487 · HTML: https://mlsanity.com/paper/2206.06487/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 45 of its 61 references that are in the map.
- First posted on arXiv 13 Jun 2022.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2007.09183/) (2020)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Spatio-Temporal Graph for Video Captioning with Knowledge Distillation](https://mlsanity.com/paper/2003.13942/) (2020)
- [Knowledge Adaptation for Efficient Semantic Segmentation](https://mlsanity.com/paper/1903.04688/) (2019)
- [ASR is all you need: cross-modal distillation for lip reading](https://mlsanity.com/paper/1911.12747/) (2019)
- [CentralNet: a Multilayer Approach for Multimodal Fusion](https://mlsanity.com/paper/1808.07275/) (2018)

## Built on by (most cited first, 30 of 44)

- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld](https://mlsanity.com/paper/2311.16714/) (2023)
- [Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning](https://mlsanity.com/paper/2303.05952/) (2023)
- [TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models](https://mlsanity.com/paper/2301.01296/) (2023)
- [Vision-Language Dataset Distillation](https://mlsanity.com/paper/2308.07545/) (2023)
- [Learning Optimal Multimodal Information Bottleneck Representations](https://mlsanity.com/paper/2505.19996/) (2025)
- [X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing](https://mlsanity.com/paper/2410.10167/) (2024)
- [TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2401.12987/) (2024)
- [Detached and Interactive Multimodal Learning](https://mlsanity.com/paper/2407.19514/) (2024)
- [COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition](https://mlsanity.com/paper/2503.07259/) (2025)
- [Learning Modality Knowledge Alignment for Cross-Modality Transfer](https://mlsanity.com/paper/2406.18864/) (2024)
- [DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learning](https://mlsanity.com/paper/2408.07080/) (2024)
- [G2D: Boosting Multimodal Learning with Gradient-Guided Distillation](https://mlsanity.com/paper/2506.21514/) (2025)
- [MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillation](https://mlsanity.com/paper/2507.07015/) (2025)
- [Distilling Cross-Modal Knowledge via Feature Disentanglement](https://mlsanity.com/paper/2511.19887/) (2025)
- [What Radio Waves Tell Us about Sleep](https://mlsanity.com/paper/2405.11739/) (2024)
- [Revisiting Cross-Modal Knowledge Distillation: A Disentanglement Approach for RGBD Semantic Segmentation](https://mlsanity.com/paper/2505.24361/) (2025)
- [C3T: Cross-modal Transfer Through Time for Sensor-based Human Activity Recognition](https://mlsanity.com/paper/2407.16803/) (2024)
- [Adaptive Conformal Guidance for Learning under Uncertainty](https://mlsanity.com/paper/2502.16736/) (2025)
- [Towards Robust Multimodal Representation: A Unified Approach with Adaptive Experts and Alignment](https://mlsanity.com/paper/2503.09498/) (2025)
- [Multi-modal Knowledge Decomposition based Online Distillation for Biomarker Prediction in Breast Cancer Histopathology](https://mlsanity.com/paper/2508.17213/) (2025)
- [Do Students Debias Like Teachers? On the Distillability of Bias Mitigation Methods](https://mlsanity.com/paper/2510.26038/) (2025)
- [Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals](https://mlsanity.com/paper/2308.08125/) (2023)
- [Balanced Multi-modal Federated Learning via Cross-Modal Infiltration](https://mlsanity.com/paper/2401.00894/) (2023)
- [Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback](https://mlsanity.com/paper/2403.11330/) (2024)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [A Generalization Theory of Cross-Modality Distillation with Contrastive Learning](https://mlsanity.com/paper/2405.03355/) (2024)
- [Non-target Divergence Hypothesis: Toward Understanding Domain Gaps in Cross-Modal Knowledge Distillation](https://mlsanity.com/paper/2409.02438/) (2024)
- [Crossmodal Knowledge Distillation with WordNet-Relaxed Text Embeddings for Robust Image Classification](https://mlsanity.com/paper/2503.24017/) (2025)
- [Uncertainty-Aware Cross-Modal Knowledge Distillation with Prototype Learning for Multimodal Brain-Computer Interfaces](https://mlsanity.com/paper/2507.13092/) (2025)

## Abstract (arXiv)

Crossmodal knowledge distillation (KD) extends traditional knowledge distillation to the area of multimodal learning and demonstrates great success in various applications. To achieve knowledge transfer across modalities, a pretrained network from one modality is adopted as the teacher to provide supervision signals to a student network learning from another modality. In contrast to the empirical success reported in prior works, the working mechanism of crossmodal KD remains a mystery. In this paper, we present a thorough understanding of crossmodal KD. We begin with two case studies and demonstrate that KD is not a universal cure in crossmodal knowledge transfer. We then present the modality Venn diagram to understand modality relationships and the modality focusing hypothesis revealing the decisive factor in the efficacy of crossmodal KD. Experimental results on 6 multimodal datasets help justify our hypothesis, diagnose failure cases, and point directions to improve crossmodal knowledge transfer in the future.