mlsanityData to 14 Oct 2026

What Makes Training Multi-Modal Classification Networks Hard?

arXiv 1905.12681 ↗First posted 29 May 2019cs.CV cs.LGcited by 344

Not on the map. Why?

Abstract (arXiv)

Consider end-to-end training of a multi-modal vs. a single-modal network on a task with multiple input modalities: the multi-modal network receives more information, so it should match or outperform its single-modal counterpart. In our experiments, however, we observe the opposite: the best single-modal network always outperforms the multi-modal network. This observation is consistent across different combinations of modalities and on different tasks and benchmarks. This paper identifies two main causes for this performance drop: first, multi-modal networks are often prone to overfitting due to increased capacity. Second, different modalities overfit and generalize at different rates, so training them jointly with a single optimization strategy is sub-optimal. We address these two problems with a technique we call Gradient Blending, which computes an optimal blend of modalities based on their overfitting behavior. We demonstrate that Gradient Blending outperforms widely-used baselines for avoiding overfitting and achieves state-of-the-art accuracy on various tasks including human action recognition, ego-centric action recognition, and acoustic event detection.

Key facts. Cited by 344 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 31; 2022: 48; 2023: 55; 2024: 68; 2025: 68; 2026: 74.

344 papers.

Sort:

By area:

  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  2. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  3. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  4. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  5. 2022Trusted Multi-View Classification with Dynamic Evidential Fusioncited by 168
  6. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
  7. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  8. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  9. 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 145
  10. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 122
  11. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  12. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 116
38 more
  1. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 99
  2. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 95
  3. 2021Trusted Multi-View Classificationcited by 94
  4. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 82
  5. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 75
  6. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 73
  7. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 73
  8. 2021SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videoscited by 64
  9. 2024ReconBoost: Boosting Can Achieve Modality Reconcilementcited by 60
  10. 2022MAViL: Masked Audio-Video Learnerscited by 59
  11. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  12. 2023Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Reviewcited by 58
  13. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  14. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  15. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  16. 2022Audiovisual Masked Autoencoderscited by 49
  17. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 49
  18. 2021VPFNet: Improving 3D Object Detection with Virtual Point based LiDAR and Stereo Data Fusioncited by 48
  19. 2023SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalizationcited by 48
  20. 2024Multimodal Prototyping for cancer survival predictioncited by 47
  21. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  22. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 44
  23. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  24. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 44
  25. 2024GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformercited by 44
  26. 2021MinkLoc++: Lidar and Monocular Image Fusion for Place Recognitioncited by 41
  27. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 41
  28. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41
  29. 2024Diagnosing and Re-learning for Balanced Multimodal Learningcited by 41
  30. 2023FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Taskscited by 38
  31. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  32. 2021Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognitioncited by 35
  33. 2022Audio-Adaptive Activity Recognition Across Video Domainscited by 35
  34. 2024360+x: A Panoptic Multi-modal Scene Understanding Datasetcited by 35
  35. 2024Classifier-guided Gradient Modulation for Enhanced Multimodal Learningcited by 35
  36. 2022A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognitioncited by 34
  37. 2025FACTR: Force-Attending Curriculum Training for Contact-Rich Policy Learningcited by 34
  38. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32

50 most cited of 344 All 344 (JSON)