mlsanityData to 7 Oct 2026

What Makes Multi-modal Learning Better than Single (Provably)

arXiv 2106.04538 ↗First posted 8 Jun 2021cs.LG cs.AIcited by 151

Not on the map. Why?

Abstract (arXiv)

The world provides us with data of multiple modalities. Intuitively, models fusing data from different modalities outperform their uni-modal counterparts, since more information is aggregated. Recently, joining the success of deep learning, there is an influential line of work on deep multi-modal learning, which has remarkable empirical results on various applications. However, theoretical justifications in this field are notably lacking. Can multi-modal learning provably perform better than uni-modal? In this paper, we answer this question under a most popular multi-modal fusion framework, which firstly encodes features from different modalities into a common latent space and seamlessly maps the latent representations into the task space. We prove that learning with multiple modalities achieves a smaller population risk than only using its subset of modalities. The main intuition is that the former has a more accurate estimate of the latent space representation. To the best of our knowledge, this is the first theoretical treatment to capture important qualitative phenomena observed in real multi-modal applications from the generalization perspective. Combining with experiment results, we show that multi-modal learning does possess an appealing formal guarantee.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Builds on 32 of its 59 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

32 of 59 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2021Convex optimizationcited by 2,479
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  11. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  12. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
20 more
  1. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  2. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  3. 2018The Sound of Pixelscited by 258
  4. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  5. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  6. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  7. 2015Cross Modal Distillation for Supervision Transfercited by 173
  8. 2020Provable Meta-Learning of Linear Representationscited by 161
  9. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  10. 2013A Survey on Multi-view Learningcited by 152
  11. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  12. 2020Deep Multimodal Fusion by Channel Exchangingcited by 117
  13. 2019ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentationcited by 105
  14. 2021Trusted Multi-View Classificationcited by 94
  15. 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 83
  16. 2018RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentationcited by 65
  17. 2020Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropiescited by 56
  18. 2017Learning Cross-Modal Deep Representations for Robust Pedestrian Detectioncited by 34
  19. 2020Large Scale Audiovisual Learning of Sounds with Weakly Labeled Datacited by 27
  20. 2020TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 18; 2023: 38; 2024: 41; 2025: 36; 2026: 17.

151 papers.

Sort:

By area:

  1. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  2. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  3. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  4. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 81
  5. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  6. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
  7. 2024IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detectioncited by 57
  8. 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 53
  9. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  10. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  11. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 48
  12. 2022Divert More Attention to Vision-Language Trackingcited by 47
38 more
  1. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  2. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 43
  3. 2023EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understandingcited by 32
  4. 2023On the Generalization of Multi-modal Contrastive Learningcited by 30
  5. 2021M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretrainingcited by 26
  6. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  7. 2023A Theory of Multimodal Learningcited by 25
  8. 2022Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Workscited by 24
  9. 2022HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalitiescited by 24
  10. 2023Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignalscited by 22
  11. 2023GiGaMAE: Generalizable Graph Masked Autoencoder via Collaborative Latent Space Reconstructioncited by 21
  12. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Methodcited by 21
  13. 2023Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Reviewcited by 20
  14. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  15. 2022Learning Multimodal Data Augmentation in Feature Spacecited by 17
  16. 2024Predictive Dynamic Fusioncited by 17
  17. 2024Quantifying and Enhancing Multi-modal Robustness with Modality Preferencecited by 16
  18. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  19. 2023MuSe-GNN: Learning Unified Gene Representation From Multimodal Biological Graph Datacited by 14
  20. 2022Multimodal Prototype-Enhanced Network for Few-Shot Action Recognitioncited by 13
  21. 2024A Survey on Integrated Sensing, Communication, and Computationcited by 12
  22. 2025CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classificationcited by 12
  23. 2025Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportioncited by 12
  24. 2023VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitionscited by 11
  25. 2023Divert More Attention to Vision-Language Object Trackingcited by 11
  26. 2024SGSeg: Enabling Text-free Inference in Language-guided Segmentation of Chest X-rays via Self-guidancecited by 11
  27. 2022Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picturecited by 10
  28. 2023kTrans: Knowledge-Aware Transformer for Binary Code Embeddingcited by 10
  29. 2023A Multi-Modal Contrastive Diffusion Model for Therapeutic Peptide Generationcited by 10
  30. 2024Understanding Retrieval-Augmented Task Adaptation for Vision-Language Modelscited by 10
  31. 2024Balanced Multi-Relational Graph Clusteringcited by 10
  32. 2024Detached and Interactive Multimodal Learningcited by 10
  33. 2024MMP: Towards Robust Multi-Modal Learning with Masked Modality Projectioncited by 10
  34. 2024Towards Modality Generalization: A Benchmark and Prospective Analysiscited by 10
  35. 2022Modality Mixer for Multi-modal Action Recognitioncited by 9
  36. 2024Beyond DAGs: A Latent Partial Causal Model for Multimodal Learningcited by 9
  37. 2024NLPrompt: Noise-Label Prompt Learning for Vision-Language Modelscited by 9
  38. 2022Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptationcited by 8

50 most cited of 151 All 151 (JSON)