mlsanityData to 7 Oct 2026

On Uni-Modal Feature Learning in Supervised Multi-Modal Learning

arXiv 2305.01233 ↗First posted 2 May 2023cs.CV cs.MMcited by 53

Not on the map. Why?

Abstract (arXiv)

We abstract the features (i.e. learned representations) of multi-modal data into 1) uni-modal features, which can be learned from uni-modal training, and 2) paired features, which can only be learned from cross-modal interactions. Multi-modal models are expected to benefit from cross-modal interactions on the basis of ensuring uni-modal feature learning. However, recent supervised multi-modal late-fusion training approaches still suffer from insufficient learning of uni-modal features on each modality. We prove that this phenomenon does hurt the model's generalization ability. To this end, we propose to choose a targeted late-fusion learning method for the given supervised multi-modal task from Uni-Modal Ensemble(UME) and the proposed Uni-Modal Teacher(UMT), according to the distribution of uni-modal and paired features. We demonstrate that, under a simple guiding strategy, we can achieve comparable results to other complex late-fusion or intermediate-fusion methods on various multi-modal datasets, including VGG-Sound, Kinetics-400, UCF101, and ModelNet40.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 2 May 2023.

Builds on

34 of 55 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  6. 2020Knowledge Distillation: A Surveycited by 1,578
  7. 2019Contrastive Representation Distillationcited by 734
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  9. 2017Look, Listen and Learncited by 452
  10. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  11. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  12. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
22 more
  1. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  2. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  3. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  4. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  5. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  6. 2023Multimodal Deep Learningcited by 257
  7. 2015Cross Modal Distillation for Supervision Transfercited by 173
  8. 2013A Survey on Multi-view Learningcited by 152
  9. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 151
  10. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  11. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  12. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  13. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  14. 2019ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentationcited by 105
  15. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 93
  16. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  17. 2018Modality Distillation with Multiple Stream Networks for Action Recognitioncited by 83
  18. 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 83
  19. 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 61
  20. 2021AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognitioncited by 36
  21. 2017Graph Distillation for Action Detection with Privileged Modalitiescited by 34
  22. 2020Large Scale Audiovisual Learning of Sounds with Weakly Labeled Datacited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 18; 2025: 18; 2026: 12.

53 papers.

Sort:

By area:

  1. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 74
  2. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
  3. 2024ReconBoost: Boosting Can Achieve Modality Reconcilementcited by 56
  4. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 48
  5. 2025DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoningcited by 44
  6. 2024Classifier-guided Gradient Modulation for Enhanced Multimodal Learningcited by 34
  7. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  8. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  9. 2024Improving Multimodal Learning with Multi-Loss Gradient Modulationcited by 16
  10. 2024Balancing Multimodal Training Through Game-Theoretic Regularizationcited by 13
  11. 2024Detached and Interactive Multimodal Learningcited by 10
  12. 2024Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learningcited by 8
38 more
  1. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  2. 2025Improving Multimodal Learning Balance and Sufficiency through Data Remixingcited by 8
  3. 2024DELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learningcited by 7
  4. 2025BalanceBenchmark: A Survey for Multimodal Imbalance Learningcited by 6
  5. 2025Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalizationcited by 6
  6. 2023What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?cited by 5
  7. 2025G2D: Boosting Multimodal Learning with Gradient-Guided Distillationcited by 5
  8. 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
  9. 2024End-to-end Semantic-centric Video-based Multimodal Affective Computingcited by 4
  10. 2024Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representationscited by 4
  11. 2024Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution Viewcited by 3
  12. 2025DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysiscited by 3
  13. 2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
  14. 2025Multimodal Representation Learning and Fusioncited by 3
  15. 2024Hybrid deep learning and physics-based neural network for programmable illumination computational microscopycited by 2
  16. 2025MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Taskscited by 2
  17. 2026Multimodal Classification via Total Correlation Maximizationcited by 2
  18. 2023Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Modelscited by 1
  19. 2023PowMix: A Versatile Regularizer for Multimodal Sentiment Analysiscited by 1
  20. 2024Enhancing ID and Text Fusion via Alternative Training in Session-based Recommendationcited by 1
  21. 2025DeepSuM: Deep Sufficient Modality Learning Frameworkcited by 1
  22. 2025Shaping Initial State Prevents Modality Competition in Multi-modal Fusion: A Two-stage Scheduling Framework via Fast Partial Information Decompositioncited by 1
  23. 2026M3D-BFS: a Multi-stage Dynamic Fusion Strategy for Sample-Adaptive Multi-Modal Brain Network Analysiscited by 1
  24. 2024Multimodal Fusion via Self-Consistent Task-Gradient Fieldscited by 0
  25. 2025Balanced Multi-view Clusteringcited by 0
  26. 2025Multi-view Video-Pose Pretraining for Operating Room Surgical Activity Recognitioncited by 0
  27. 2025Rebalanced Multimodal Learning with Data-aware Unimodal Samplingcited by 0
  28. 2025EgoVIS@CVPR: PAIR-Net: Enhancing Egocentric Speaker Detection via Pretrained Audio-Visual Fusion and Alignment Losscited by 0
  29. 2025Partitioner Guided Modal Learning Frameworkcited by 0
  30. 2025Balanced Multimodal Learning: An Unidirectional Dynamic Interaction Perspectivecited by 0
  31. 2025Mitigating Sample-Level Imbalance via Probabilistic Separation for Adaptive Multimodal Fusioncited by 0
  32. 2026SynLeaF: A Dual-Stage Multimodal Fusion Framework for Synthetic Lethality Prediction Across Pan- and Single-Cancer Contextscited by 0
  33. 2026Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognitioncited by 0
  34. 2026Boosting Multimodal Federated Learning via Chained Modality Optimizationcited by 0
  35. 2026ASR-Agnostic Multimodal Spectrotemporal Modeling for Early Dementia Detectioncited by 0
  36. 2026Uncertainty-aware damage identification in short-span bridges via physics-informed variational autoencodercited by 0
  37. 2026From Cellular Responses to Pharmacological Domains: Multimodal Zero-Shot Drug Representation Learningcited by 0
  38. 2026CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgerycited by 0

50 most cited of 53 All 53 (JSON)