mlsanityData to 7 Oct 2026

Enhanced Multimodal Representation Learning with Cross-modal KD

arXiv 2306.07646 ↗First posted 13 Jun 2023cs.CV cs.MMcited by 11

Not on the map. Why?

Abstract (arXiv)

This paper explores the tasks of leveraging auxiliary modalities which are only available at training to enhance multimodal representation learning through cross-modal Knowledge Distillation (KD). The widely adopted mutual information maximization-based objective leads to a short-cut solution of the weak teacher, i.e., achieving the maximum mutual information by simply making the teacher model as weak as the student model. To prevent such a weak solution, we introduce an additional objective term, i.e., the mutual information between the teacher and the auxiliary modality model. Besides, to narrow down the information gap between the student and teacher, we further propose to minimize the conditional entropy of the teacher given the student. Novel training schemes based on contrastive learning and adversarial learning are designed to optimize the mutual information and the conditional entropy, respectively. Experimental results on three popular multimodal benchmark datasets have shown that the proposed method outperforms a range of state-of-the-art approaches for video recognition, video retrieval and emotion classification.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

37 of 52 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2014Conditional Generative Adversarial Netscited by 2,152
  7. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  11. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  12. 2016Deep Variational Information Bottleneckcited by 915
25 more
  1. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  2. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  3. 2019Contrastive Representation Distillationcited by 734
  4. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  5. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  6. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  7. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  8. 2017Teacher-Student Curriculum Learningcited by 197
  9. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  10. 2019Variational Bayesian Optimal Experimental Designcited by 116
  11. 2018Modality Distillation with Multiple Stream Networks for Action Recognitioncited by 83
  12. 2021Knowledge distillation from multi-modal to mono-modal segmentation networkscited by 68
  13. 2020Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusioncited by 67
  14. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  15. 2016Generative Multi-Adversarial Networkscited by 60
  16. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  17. 2021Contrastive Multimodal Fusion with TupleInfoNCEcited by 43
  18. 2021Complementary Relation Contrastive Distillationcited by 42
  19. 2022Geometric Multimodal Contrastive Representation Learningcited by 40
  20. 2018Improving Spatiotemporal Self-Supervision by Deep Reinforcement Learningcited by 35
  21. 2021Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detectioncited by 32
  22. 2018Cross-Modality Synthesis from CT to PET using FCN and GAN Networks for Improved Automated Lesion Detectioncited by 29
  23. 2019Privileged Features Distillation at Taobao Recommendationscited by 20
  24. 2017Stabilizing GAN Training with Multiple Random Projectionscited by 15
  25. 2018On-line Adaptative Curriculum Learning for GANscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 7; 2026: 1.

11 papers.

  1. 2023Combating Representation Learning Disparity with Geometric Harmonizationcited by 11
  2. 2025DGS-Net: Distillation-Guided Gradient Surgery for CLIP Fine-Tuning in AI-Generated Image Detectioncited by 4
  3. 2025Differential-informed Sample Selection Accelerates Multimodal Contrastive Learningcited by 3
  4. 2025Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learningcited by 3
  5. 2024C3T: Cross-modal Transfer Through Time for Sensor-based Human Activity Recognitioncited by 2
  6. 2025TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimizationcited by 2
  7. 2025Learning to Match Unpaired Data with Minimum Entropy Couplingcited by 2
  8. 2025Multimodal Negative Learningcited by 2
  9. 2024Non-target Divergence Hypothesis: Toward Understanding Domain Gaps in Cross-Modal Knowledge Distillationcited by 1
  10. 2025Enriched text-guided variational multimodal knowledge distillation network (VMD) for automated diagnosis of plaque vulnerability in 3D carotid artery MRIcited by 0
  11. 2026FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learningcited by 0