mlsanityData to 7 Oct 2026

Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)

arXiv 2203.12221 ↗First posted 23 Mar 2022cs.LGcited by 115

Not on the map. Why?

Abstract (arXiv)

Despite the remarkable success of deep multi-modal learning in practice, it has not been well-explained in theory. Recently, it has been observed that the best uni-modal network outperforms the jointly trained multi-modal network, which is counter-intuitive since multiple signals generally bring more information. This work provides a theoretical explanation for the emergence of such performance gap in neural networks for the prevalent joint training framework. Based on a simplified data distribution that captures the realistic property of multi-modal data, we prove that for the multi-modal late-fusion network with (smoothed) ReLU activation trained jointly by gradient descent, different modalities will compete with each other. The encoder networks will learn only a subset of modalities. We refer to this phenomenon as modality competition. The losing modalities, which fail to be discovered, are the origins where the sub-optimality of joint training comes from. Experimentally, we illustrate that modality competition matches the intrinsic behavior of late-fusion joint training.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 37 of its 64 references that are in the map. First posted on arXiv 23 Mar 2022.

Builds on

37 of 64 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  8. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  11. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  12. 2015Listen, Attend and Spellcited by 542
25 more
  1. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  2. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  3. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  4. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  5. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  6. 2015Cross Modal Distillation for Supervision Transfercited by 173
  7. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  8. 2013A Survey on Multi-view Learningcited by 152
  9. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 151
  10. 2019The Sound of Motionscited by 145
  11. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  12. 2019Audio-Visual Scene-Aware Dialogcited by 122
  13. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  14. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  15. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  16. 2021M6: A Chinese Multimodal Pretrainercited by 91
  17. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  18. 2018RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentationcited by 65
  19. 2020Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropiescited by 56
  20. 2021AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognitioncited by 36
  21. 2018Learn to Combine Modalities in Multimodal Deep Learningcited by 29
  22. 2020TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learningcited by 19
  23. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  24. 2014Learning Word Representations with Hierarchical Sparse Codingcited by 4
  25. 2016Multimodal Sparse Coding for Event Detectioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 21; 2024: 30; 2025: 34; 2026: 27.

115 papers.

Sort:

By area:

  1. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  2. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  3. 2023In-Context Convergence of Transformerscited by 112
  4. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 81
  5. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 74
  6. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
  7. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 71
  8. 2024ReconBoost: Boosting Can Achieve Modality Reconcilementcited by 56
  9. 2022Multimodal learning with graphscited by 54
  10. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  11. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 48
  12. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 43
38 more
  1. 2024Diagnosing and Re-learning for Balanced Multimodal Learningcited by 40
  2. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  3. 2024Classifier-guided Gradient Modulation for Enhanced Multimodal Learningcited by 34
  4. 2024On-the-fly Modulation for Balanced Multimodal Learningcited by 32
  5. 2023Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicinecited by 24
  6. 2023Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Reviewcited by 20
  7. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  8. 2025Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisitioncited by 18
  9. 2024Quantifying and Enhancing Multi-modal Robustness with Modality Preferencecited by 16
  10. 2024Improving Multimodal Learning with Multi-Loss Gradient Modulationcited by 16
  11. 2024Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworkscited by 15
  12. 2023UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learningcited by 13
  13. 2024Balancing Multimodal Training Through Game-Theoretic Regularizationcited by 13
  14. 2025Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportioncited by 12
  15. 2024Gradient-Guided Modality Decoupling for Missing-Modality Robustnesscited by 11
  16. 2023CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalitiescited by 10
  17. 2024Mitigating Noisy Correspondence by Geometrical Structure Consistency Learningcited by 10
  18. 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directionscited by 10
  19. 2024Detached and Interactive Multimodal Learningcited by 10
  20. 2024Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillationcited by 10
  21. 2022Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonizationcited by 9
  22. 2023Towards Balanced Active Learning for Multimodal Classificationcited by 9
  23. 2025See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Biascited by 9
  24. 2025Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representationscited by 9
  25. 2025Improving Multimodal Learning Balance and Sufficiency through Data Remixingcited by 8
  26. 2023Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Groundingcited by 7
  27. 2024Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropoutcited by 7
  28. 2024MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Modelcited by 7
  29. 2026Unbiased Dynamic Multimodal Fusioncited by 7
  30. 2023Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selectioncited by 6
  31. 2025SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Modelscited by 6
  32. 2023What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?cited by 5
  33. 2024Rebalanced Vision-Language Retrieval Considering Structure-Aware Distillationcited by 5
  34. 2025G2D: Boosting Multimodal Learning with Gradient-Guided Distillationcited by 5
  35. 2025Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoderscited by 5
  36. 2025The More, the Merrier: Contrastive Fusion for Higher-Order Multimodal Alignmentcited by 5
  37. 2023MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learningcited by 4
  38. 2023Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectivescited by 4

50 most cited of 115 All 115 (JSON)