mlsanityData to 7 Oct 2026

Boosting Multi-modal Model Performance with Adaptive Gradient Modulation

arXiv 2308.07686 ↗First posted 15 Aug 2023cs.CV cs.AIcited by 71

Not on the map. Why?

Abstract (arXiv)

While the field of multi-modal learning keeps growing fast, the deficiency of the standard joint training paradigm has become clear through recent studies. They attribute the sub-optimal performance of the jointly trained model to the modality competition phenomenon. Existing works attempt to improve the jointly trained model by modulating the training process. Despite their effectiveness, those methods can only apply to late fusion models. More importantly, the mechanism of the modality competition remains unexplored. In this paper, we first propose an adaptive gradient modulation method that can boost the performance of multi-modal models with various fusion strategies. Extensive experiments show that our method surpasses all existing modulation methods. Furthermore, to have a quantitative understanding of the modality competition and the mechanism behind the effectiveness of our modulation method, we introduce a novel metric to measure the competition strength. This metric is built on the mono-modal concept, a function that is designed to represent the competition-less state of a modality. Through systematic investigation, our results confirm the intuition that the modulation encourages the model to rely on the more informative modality. In addition, we find that the jointly trained model typically has a preferred modality on which the competition is weaker than other modalities. However, this preferred modality need not dominate others. Our code will be available at https://github.com/lihong2303/AGM_ICCV2023.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 15 Aug 2023.

Builds on

26 of 35 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  5. 2017The Kinetics Human Action Video Datasetcited by 1,894
  6. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  7. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  8. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  9. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  10. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  11. 2013Maxout Networkscited by 186
  12. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
14 more
  1. 2022Trusted Multi-View Classification with Dynamic Evidential Fusioncited by 167
  2. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  3. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
  4. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  5. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  6. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  7. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  8. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  9. 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 61
  10. 2020A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysiscited by 46
  11. 2021Perceptual Score: What Data Modalities Does Your Model Perceive?cited by 35
  12. 2022Uncertainty-Aware Multi-View Representation Learningcited by 29
  13. 2018Computationally Efficient Measures of Internal Neuron Importancecited by 19
  14. 2022SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalitiescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 32; 2026: 29.

71 papers.

Sort:

By area:

  1. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
  2. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 43
  3. 2024Diagnosing and Re-learning for Balanced Multimodal Learningcited by 40
  4. 2024Classifier-guided Gradient Modulation for Enhanced Multimodal Learningcited by 34
  5. 2025Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisitioncited by 18
  6. 2024Improving Multimodal Learning with Multi-Loss Gradient Modulationcited by 16
  7. 2025Improving Multimodal Learning via Imbalanced Learningcited by 16
  8. 2024Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworkscited by 15
  9. 2025Boosting Multimodal Learning via Disentangled Gradient Learningcited by 15
  10. 2024Balancing Multimodal Training Through Game-Theoretic Regularizationcited by 13
  11. 2025Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportioncited by 12
  12. 2025Towards Balanced Multi-Modal Learning in 3D Human Pose Estimationcited by 11
38 more
  1. 2025See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Biascited by 9
  2. 2025Improving Multimodal Learning Balance and Sufficiency through Data Remixingcited by 8
  3. 2023Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selectioncited by 6
  4. 2025BalanceBenchmark: A Survey for Multimodal Imbalance Learningcited by 6
  5. 2025Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learningcited by 5
  6. 2025G2D: Boosting Multimodal Learning with Gradient-Guided Distillationcited by 5
  7. 2026CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learningcited by 5
  8. 2024Multimodal Classification via Modal-Aware Interactive Enhancementcited by 4
  9. 2025TiCAL:Typicality-Based Consistency-Aware Learning for Multimodal Emotion Recognitioncited by 4
  10. 2025Multimodal Representation Learning and Fusioncited by 3
  11. 2025SAMSOD: Rethinking SAM Optimization for RGB-T Salient Object Detectioncited by 3
  12. 2026Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integrationcited by 3
  13. 2025Balance-aware Sequence Sampling Makes Multi-modal Learning Bettercited by 2
  14. 2025Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learningcited by 2
  15. 2025MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learningcited by 2
  16. 2025Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learningcited by 2
  17. 2026Multimodal Classification via Total Correlation Maximizationcited by 2
  18. 2025Decoding Visual Neural Representations by Multimodal with Dynamic Balancingcited by 1
  19. 2025Shaping Initial State Prevents Modality Competition in Multi-modal Fusion: A Two-stage Scheduling Framework via Fast Partial Information Decompositioncited by 1
  20. 2025Revisit Modality Imbalance at the Decision Layercited by 1
  21. 2026Modality Dominance-Aware Optimization for Embodied RGB-Infrared Perceptioncited by 1
  22. 2026Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusioncited by 1
  23. 2026Balancing Multimodal Domain Generalization via Gradient Modulation and Projectioncited by 1
  24. 2026BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Ratescited by 1
  25. 2026PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritizationcited by 1
  26. 2023Dissecting RGB-D Learning for Improved Multi-modal Fusioncited by 0
  27. 2025Enhancing Scene Classification in Cloudy Image Scenarios: A Collaborative Transfer Method with Information Regulation Mechanism using Optical Cloud-Covered and SAR Remote Sensing Imagescited by 0
  28. 2025Rebalanced Multimodal Learning with Data-aware Unimodal Samplingcited by 0
  29. 2025CMD-HAR: Cross-Modal Disentanglement for Wearable Human Activity Recognitioncited by 0
  30. 2025EgoVIS@CVPR: PAIR-Net: Enhancing Egocentric Speaker Detection via Pretrained Audio-Visual Fusion and Alignment Losscited by 0
  31. 2025DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentationcited by 0
  32. 2025Confidence-driven Gradient Modulation for Multimodal Human Activity Recognition: A Dynamic Contrastive Dual-Path Learning Approachcited by 0
  33. 2025AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learningcited by 0
  34. 2025Balanced Multimodal Learning: An Unidirectional Dynamic Interaction Perspectivecited by 0
  35. 2025Preserving Cross-Modal Stability for Visual Unlearning in Multimodal Scenarioscited by 0
  36. 2025MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulationcited by 0
  37. 2025Mitigating Sample-Level Imbalance via Probabilistic Separation for Adaptive Multimodal Fusioncited by 0
  38. 2025Balancing Multi-modal Sensor Learning via Multi-objective Optimizationcited by 0

50 most cited of 71 All 71 (JSON)