mlsanityData to 7 Oct 2026

Balanced Multimodal Learning via On-the-fly Gradient Modulation

arXiv 2203.15332 ↗First posted 29 Mar 2022cs.CV cs.AIcited by 259

Not on the map. Why?

Abstract (arXiv)

Multimodal learning helps to comprehensively understand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance, but we actually find that they are not fully exploited even when the multimodal model outperforms its uni-modal counterpart. Specifically, in this paper we point out that existing multimodal discriminative models, in which uniform objective is designed for all modalities, could remain under-optimized uni-modal representations, caused by another dominated modality in some scenarios, e.g., sound in blowing wind event, vision in drawing picture event, etc. To alleviate this optimization imbalance, we propose on-the-fly gradient modulation to adaptively control the optimization of each modality, via monitoring the discrepancy of their contribution towards the learning objective. Further, an extra Gaussian noise that changes dynamically is introduced to avoid possible generalization drop caused by gradient modulation. As a result, we achieve considerable improvement over common fusion methods on different multimodal tasks, and this simple strategy can also boost existing multimodal methods, which illustrates its efficacy and versatility. The source code is available at urlhttps://github.com/GeWu-Lab/OGM-GECVPR2022.

Key facts. Cited by 259 papers in the map (data through 7 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 29 Mar 2022.

Builds on

32 of 51 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2018SlowFast Networks for Video Recognitioncited by 1,862
  6. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  7. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  9. 2017Look, Listen and Learncited by 452
  10. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  11. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  12. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
20 more
  1. 2017Three Factors Influencing Minima in SGDcited by 270
  2. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  3. 2018The Sound of Pixelscited by 258
  4. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  5. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  6. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  7. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  8. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  9. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  10. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  11. 2020Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learningcited by 63
  12. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 40
  13. 2020x-vectors meet emotions: A study on dependencies between emotion and speaker recognitioncited by 31
  14. 2020Temporal Relational Modeling with Self-Supervision for Action Segmentationcited by 29
  15. 2020Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgettingcited by 28
  16. 2019Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videoscited by 27
  17. 2020Speech2Action: Cross-modal Supervision for Action Recognitioncited by 26
  18. 2020On Modality Bias in the TVQA Datasetcited by 24
  19. 2020Improving Multimodal Accuracy Through Modality Pre-training and Attentioncited by 15
  20. 2018Dense Multimodal Fusion for Hierarchically Joint Representationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 32; 2024: 51; 2025: 78; 2026: 92.

259 papers.

Sort:

By area:

  1. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 114
  2. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 81
  3. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 74
  4. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
  5. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 71
  6. 2024ReconBoost: Boosting Can Achieve Modality Reconcilementcited by 56
  7. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  8. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
  9. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 48
  10. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  11. 2022Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Modulecited by 43
  12. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 43
38 more
  1. 2024Diagnosing and Re-learning for Balanced Multimodal Learningcited by 40
  2. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  3. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  4. 2023Decoupling Common and Unique Representations for Multimodal Self-supervised Learningcited by 36
  5. 2024360+x: A Panoptic Multi-modal Scene Understanding Datasetcited by 35
  6. 2024Classifier-guided Gradient Modulation for Enhanced Multimodal Learningcited by 34
  7. 2023Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentationcited by 33
  8. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  9. 2024On-the-fly Modulation for Balanced Multimodal Learningcited by 32
  10. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  11. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  12. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  13. 2022Multimodal Analogical Reasoning over Knowledge Graphscited by 21
  14. 2025Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoningcited by 20
  15. 2024Fairness and Bias in Multimodal AI: A Surveycited by 19
  16. 2023UniCat: Crafting a Stronger Fusion Baseline for Multimodal Re-Identificationcited by 18
  17. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  18. 2025Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisitioncited by 18
  19. 2024AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesiscited by 17
  20. 2024Predictive Dynamic Fusioncited by 17
  21. 2024Robust Multimodal Learning via Representation Decouplingcited by 17
  22. 2024PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Ratescited by 17
  23. 2024Quantifying and Enhancing Multi-modal Robustness with Modality Preferencecited by 16
  24. 2024Improving Multimodal Learning with Multi-Loss Gradient Modulationcited by 16
  25. 2025DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learningcited by 16
  26. 2025Improving Multimodal Learning via Imbalanced Learningcited by 16
  27. 2023Continual Multimodal Knowledge Graph Constructioncited by 15
  28. 2024Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworkscited by 15
  29. 2025Learning Optimal Multimodal Information Bottleneck Representationscited by 15
  30. 2025Boosting Multimodal Learning via Disentangled Gradient Learningcited by 15
  31. 2023MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimizationcited by 14
  32. 2023RedCore: Relative Advantage Aware Cross-modal Representation Learning for Missing Modalities with Imbalanced Missing Ratescited by 14
  33. 2023FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibrationcited by 13
  34. 2023Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognitioncited by 13
  35. 2024Balancing Multimodal Training Through Game-Theoretic Regularizationcited by 13
  36. 2025Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportioncited by 12
  37. 2023Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsingcited by 11
  38. 2024Gradient-Guided Modality Decoupling for Missing-Modality Robustnesscited by 11

50 most cited of 259 All 259 (JSON)