mlsanityData to 7 Oct 2026

Multimodal Knowledge Expansion

arXiv 2103.14431 ↗First posted 26 Mar 2021cs.CV cs.LG cs.MMcited by 26

Not on the map. Why?

Abstract (arXiv)

The popularity of multimodal sensors and the accessibility of the Internet have brought us a massive amount of unlabeled multimodal data. Since existing datasets and well-trained models are primarily unimodal, the modality gap between a unimodal network and unlabeled multimodal data poses an interesting problem: how to transfer a pre-trained unimodal network to perform the same task on unlabeled multimodal data? In this work, we propose multimodal knowledge expansion (MKE), a knowledge distillation-based framework to effectively utilize multimodal data without requiring labels. Opposite to traditional knowledge distillation, where the student is designed to be lightweight and inferior to the teacher, we observe that a multimodal student model consistently denoises pseudo labels and generalizes better than its teacher. Extensive experiments on four tasks and different modalities verify this finding. Furthermore, we connect the mechanism of MKE to semi-supervised learning and offer both empirical and theoretical explanations to understand the denoising capability of a multimodal student.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 26 Mar 2021.

Builds on

41 of 53 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  5. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  7. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  8. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  9. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  10. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  11. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  12. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
29 more
  1. 2019Similarity-Preserving Knowledge Distillationcited by 507
  2. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  3. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  4. 2017Look, Listen and Learncited by 452
  5. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  6. 2019Confidence Regularized Self-Trainingcited by 418
  7. 2020Meta Pseudo Labelscited by 372
  8. 2020Rethinking Pre-training and Self-trainingcited by 352
  9. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  10. 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 313
  11. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  12. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  13. 2014Learning with Pseudo-Ensemblescited by 241
  14. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  15. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  16. 2015Cross Modal Distillation for Supervision Transfercited by 173
  17. 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
  18. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 120
  19. 2020Deep Multimodal Fusion by Channel Exchangingcited by 117
  20. 2016Ambient Sound Provides Supervision for Visual Learningcited by 77
  21. 2018Self-Supervised Model Adaptation for Multimodal Semantic Segmentationcited by 73
  22. 2017End-to-End Multimodal Emotion Recognition using Deep Neural Networkscited by 69
  23. 2018Learnable PINs: Cross-Modal Embeddings for Person Identitycited by 60
  24. 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50
  25. 2018Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modelingcited by 39
  26. 2020Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentationcited by 38
  27. 2020Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledgecited by 35
  28. 2020Multimodal Categorization of Crisis Events in Social Mediacited by 34
  29. 2020TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 4; 2024: 4; 2025: 7; 2026: 4.

26 papers.

Sort:

  1. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  2. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  3. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  4. 2021Co-advise: Cross Inductive Bias Distillationcited by 40
  5. 2023Multimodal Distillation for Egocentric Action Recognitioncited by 28
  6. 2022Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detectioncited by 26
  7. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  8. 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Modelscited by 24
  9. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  10. 2024Segment Anything with Multiple Modalitiescited by 12
  11. 2022Learning from Multiple Annotator Noisy Labels via Sample-wise Label Fusioncited by 10
  12. 2025COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognitioncited by 8
14 more
  1. 2025MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillationcited by 5
  2. 2022Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularizationcited by 2
  3. 2025MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Taskscited by 2
  4. 2025A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Featurescited by 2
  5. 2024Ranking Distillation for Open-Ended Video Question Answering with Insufficient Labelscited by 1
  6. 2024A Generalization Theory of Cross-Modality Distillation with Contrastive Learningcited by 1
  7. 2025BioX-Bridge: Model Bridging for Unsupervised Cross-Modal Knowledge Transfer across Biosignalscited by 1
  8. 2023LIB-KD: Teaching Inductive Bias for Efficient Vision Transformer Distillation and Compressioncited by 0
  9. 2025SciceVPR: Stable Cross-Image Correlation Enhanced Model for Visual Place Recognitioncited by 0
  10. 2025Dual-frequency Selected Knowledge Distillation with Statistical-based Sample Rectification for PolSAR Image Classificationcited by 0
  11. 2026Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidancecited by 0
  12. 2026Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithmcited by 0
  13. 2026Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensorscited by 0
  14. 2026Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledgecited by 0