mlsanityData to 14 Oct 2026

A Theory of Multimodal Learning

arXiv 2309.12458 ↗First posted 21 Sep 2023cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Human perception of the empirical world involves recognizing the diverse appearances, or 'modalities', of underlying objects. Despite the longstanding consideration of this perspective in philosophy and cognitive science, the study of multimodality remains relatively under-explored within the field of machine learning. Nevertheless, current studies of multimodal machine learning are limited to empirical practices, lacking theoretical foundations beyond heuristic arguments. An intriguing finding from the practice of multimodal learning is that a model trained on multiple modalities can outperform a finely-tuned unimodal model, even on unimodal tasks. This paper provides a theoretical framework that explains this phenomenon, by studying generalization properties of multimodal learning algorithms. We demonstrate that multimodal learning allows for a superior generalization bound compared to unimodal learning, up to a factor of O(√n), where n represents the sample size. Such advantage occurs when both connection and heterogeneity exist between the modalities.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 21 of its 42 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

21 of 42 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Generative Adversarial Networkscited by 5,627
  6. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  7. 2022A Generalist Agentcited by 808
  8. 2016Generative Adversarial Text to Image Synthesiscited by 735
  9. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  10. 2023Multimodal Deep Learningcited by 258
  11. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  12. 2015The Benefit of Multitask Representation Learningcited by 195
9 more
  1. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 191
  2. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  3. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
  4. 2013A PAC-Bayesian bound for Lifelong Learningcited by 69
  5. 2015Text to 3D Scene Generation with Rich Lexical Groundingcited by 36
  6. 2023On the Provable Advantage of Unsupervised Pretrainingcited by 22
  7. 2020TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learningcited by 19
  8. 2014A chain rule for the expected suprema of Gaussian processescited by 13
  9. 2023On the Importance of Contrastive Loss in Multimodal Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 12; 2026: 7.

25 papers.

Sort:

By area:

  1. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  2. 2025Principled Multimodal Representation Learningcited by 19
  3. 2024MMP: Towards Robust Multi-Modal Learning with Masked Modality Projectioncited by 10
  4. 2024Towards Modality Generalization: A Benchmark and Prospective Analysiscited by 10
  5. 2025Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variablescited by 7
  6. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distancecited by 4
  7. 2025Multimodal Representation Learning and Fusioncited by 3
  8. 2025Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensionalcited by 3
  9. 2025Calibrated Multimodal Representation Learning with Missing Modalitiescited by 3
  10. 2025Learning to Match Unpaired Data with Minimum Entropy Couplingcited by 2
  11. 2023On the Computational Benefit of Multimodal Learningcited by 1
  12. 2024An Art-centric perspective on AI-based content moderation of nuditycited by 1
13 more
  1. 2025Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deploymentcited by 1
  2. 2025Toward Inherently Robust VLMs Against Visual Perception Attackscited by 1
  3. 2025Distributed Information Bottleneck Theory for Multi-Modal Task-Aware Semantic Communicationcited by 1
  4. 2025Sigma: The Key for Vision-Language-Action Models toward Telepathic Alignmentcited by 1
  5. 2026Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervalscited by 1
  6. 2026SplAttN: Bridging 2D and 3D with Gaussian Soft Splatting and Attention for Point Cloud Completioncited by 1
  7. 2026When to Align, When to Predict: A Phase Diagram for Multimodal Learningcited by 1
  8. 2024On Stronger Computational Separations Between Multimodal and Unimodal Machine Learningcited by 0
  9. 2024HyperMM : Robust Multimodal Learning with Varying-sized Inputscited by 0
  10. 2025Balancing Multi-modal Sensor Learning via Multi-objective Optimizationcited by 0
  11. 2026Quantifying Multimodal Capabilities: Formal Generalization Guarantees in Pairwise Metric Learningcited by 0
  12. 2026Uncertainty-aware damage identification in short-span bridges via physics-informed variational autoencodercited by 0
  13. 2026Hub-Spectral Activation of Latent Multimodal Knowledgecited by 0