mlsanityData to 7 Oct 2026

The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation

arXiv 2206.06487 ↗First posted 13 Jun 2022cs.CV cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Crossmodal knowledge distillation (KD) extends traditional knowledge distillation to the area of multimodal learning and demonstrates great success in various applications. To achieve knowledge transfer across modalities, a pretrained network from one modality is adopted as the teacher to provide supervision signals to a student network learning from another modality. In contrast to the empirical success reported in prior works, the working mechanism of crossmodal KD remains a mystery. In this paper, we present a thorough understanding of crossmodal KD. We begin with two case studies and demonstrate that KD is not a universal cure in crossmodal knowledge transfer. We then present the modality Venn diagram to understand modality relationships and the modality focusing hypothesis revealing the decisive factor in the efficacy of crossmodal KD. Experimental results on 6 multimodal datasets help justify our hypothesis, diagnose failure cases, and point directions to improve crossmodal knowledge transfer in the future.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 45 of its 61 references that are in the map. First posted on arXiv 13 Jun 2022.

Builds on

45 of 61 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  5. 2020Knowledge Distillation: A Surveycited by 1,578
  6. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  7. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  8. 2019Relational Knowledge Distillationcited by 873
  9. 2019Contrastive Representation Distillationcited by 734
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  11. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  12. 2019Similarity-Preserving Knowledge Distillationcited by 507
33 more
  1. 2019On the Efficacy of Knowledge Distillationcited by 377
  2. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  3. 2019Correlation Congruence for Knowledge Distillationcited by 280
  4. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  5. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  6. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  7. 2017Gated Multimodal Units for Information Fusioncited by 244
  8. 2015Unifying distillation and privileged informationcited by 218
  9. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  10. 2015Cross Modal Distillation for Supervision Transfercited by 173
  11. 2021Towards Understanding Knowledge Distillationcited by 158
  12. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  13. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 120
  14. 2020Deep Multimodal Fusion by Channel Exchangingcited by 117
  15. 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
  16. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  17. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  18. 2019ASR is all you need: cross-modal distillation for lip readingcited by 85
  19. 2018Modality Distillation with Multiple Stream Networks for Action Recognitioncited by 83
  20. 2020Understanding and Improving Knowledge Distillationcited by 83
  21. 2021Knowledge distillation from multi-modal to mono-modal segmentation networkscited by 68
  22. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65
  23. 2018Online Model Distillation for Efficient Video Inferencecited by 60
  24. 2019Cross-modal knowledge distillation for action recognitioncited by 44
  25. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  26. 2021Co-advise: Cross Inductive Bias Distillationcited by 40
  27. 2020Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillationcited by 38
  28. 2017Graph Distillation for Action Detection with Privileged Modalitiescited by 34
  29. 2020Feature Importance Ranking for Deep Learningcited by 31
  30. 2021Multimodal Knowledge Expansioncited by 26
  31. 2016Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networkscited by 24
  32. 2020TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learningcited by 19
  33. 2022Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 11; 2025: 15; 2026: 11.

44 papers.

Sort:

By area:

  1. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  2. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  3. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  4. 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Modelscited by 24
  5. 2023Vision-Language Dataset Distillationcited by 18
  6. 2025Learning Optimal Multimodal Information Bottleneck Representationscited by 15
  7. 2024TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversationcited by 12
  8. 2024X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensingcited by 12
  9. 2024Detached and Interactive Multimodal Learningcited by 10
  10. 2024Learning Modality Knowledge Alignment for Cross-Modality Transfercited by 8
  11. 2025COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognitioncited by 8
  12. 2024DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learningcited by 6
32 more
  1. 2025G2D: Boosting Multimodal Learning with Gradient-Guided Distillationcited by 5
  2. 2025MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillationcited by 5
  3. 2025Distilling Cross-Modal Knowledge via Feature Disentanglementcited by 5
  4. 2024What Radio Waves Tell Us about Sleepcited by 4
  5. 2025Revisiting Cross-Modal Knowledge Distillation: A Disentanglement Approach for RGBD Semantic Segmentationcited by 4
  6. 2024C3T: Cross-modal Transfer Through Time for Sensor-based Human Activity Recognitioncited by 2
  7. 2025Adaptive Conformal Guidance for Learning under Uncertaintycited by 2
  8. 2025Towards Robust Multimodal Representation: A Unified Approach with Adaptive Experts and Alignmentcited by 2
  9. 2025Multi-modal Knowledge Decomposition based Online Distillation for Biomarker Prediction in Breast Cancer Histopathologycited by 2
  10. 2025Do Students Debias Like Teachers? On the Distillability of Bias Mitigation Methodscited by 2
  11. 2023Radio2Text: Streaming Speech Recognition Using mmWave Radio Signalscited by 1
  12. 2023Balanced Multi-modal Federated Learning via Cross-Modal Infiltrationcited by 1
  13. 2024Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedbackcited by 1
  14. 2024A Generalization Theory of Cross-Modality Distillation with Contrastive Learningcited by 1
  15. 2024Non-target Divergence Hypothesis: Toward Understanding Domain Gaps in Cross-Modal Knowledge Distillationcited by 1
  16. 2025Crossmodal Knowledge Distillation with WordNet-Relaxed Text Embeddings for Robust Image Classificationcited by 1
  17. 2025Uncertainty-Aware Cross-Modal Knowledge Distillation with Prototype Learning for Multimodal Brain-Computer Interfacescited by 1
  18. 2026LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillationcited by 1
  19. 2026Harmonized Tabular-Image Fusion via Gradient-Aligned Alternating Learningcited by 1
  20. 2026Evidence of Absence: Cross-Modal Abductive Risk Perception to Sustain World Models When Vision Failscited by 1
  21. 2024Foundations of Multisensory Artificial Intelligencecited by 0
  22. 2025Domain Adaptation-Based Crossmodal Knowledge Distillation for 3D Semantic Segmentationcited by 0
  23. 2025Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learningcited by 0
  24. 2025Uncertainty-Resilient Multimodal Learning via Consistency-Guided Cross-Modal Transfercited by 0
  25. 2026MuDD: A Multimodal Deception Detection Dataset and GSR-Guided Progressive Distillation for Non-Contact Deception Detectioncited by 0
  26. 2026Purify-then-Align: Towards Robust Human Sensing under Modality Missing with Knowledge Distillation from Noisy Multimodal Teachercited by 0
  27. 2026Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithmcited by 0
  28. 2026Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledgecited by 0
  29. 2026Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probingcited by 0
  30. 2026Cross-Modal Knowledge Distillation for Acoustic Pedestrian Detectioncited by 0
  31. 2026Learning Through Game: Skewed Transfer of Tabular Knowledge to Strengthen Image Modelcited by 0
  32. 2026Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Featurescited by 0