mlsanityData to 7 Oct 2026

Classifier-guided Gradient Modulation for Enhanced Multimodal Learning

arXiv 2411.01409 ↗First posted 3 Nov 2024cs.LG cs.CL cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

Multimodal learning has developed very fast in recent years. However, during the multimodal training process, the model tends to rely on only one modality based on which it could learn faster, thus leading to inadequate use of other modalities. Existing methods to balance the training process always have some limitations on the loss functions, optimizers and the number of modalities and only consider modulating the magnitude of the gradients while ignoring the directions of the gradients. To solve these problems, in this paper, we present a novel method to balance multimodal learning with Classifier-Guided Gradient Modulation (CGGM), considering both the magnitude and directions of the gradients. We conduct extensive experiments on four multimodal datasets: UPMC-Food 101, CMU-MOSI, IEMOCAP and BraTS 2021, covering classification, regression and segmentation tasks. The results show that CGGM outperforms all the baselines and other state-of-the-art methods consistently, demonstrating its effectiveness and versatility. Our code is available at https://github.com/zrguo/CGGM.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 3 Nov 2024.

Builds on

22 of 30 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  5. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  6. 2021The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classificationcited by 507
  7. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  8. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  9. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  10. 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 128
  11. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  12. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 114
10 more
  1. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  2. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  3. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  4. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 81
  5. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 71
  6. 2024ReconBoost: Boosting Can Achieve Modality Reconcilementcited by 56
  7. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  8. 2018Multimodal Language Analysis with Recurrent Multistage Fusioncited by 43
  9. 2024Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognitioncited by 26
  10. 2023Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 18.

34 papers.

Sort:

By area:

  1. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 23
  2. 2025DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learningcited by 16
  3. 2025Towards Balanced Multi-Modal Learning in 3D Human Pose Estimationcited by 11
  4. 2025Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noisescited by 9
  5. 2025Efficient Prompting for Continual Adaptation to Missing Modalitiescited by 8
  6. 2025Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalizationcited by 6
  7. 2026Tri-Subspaces Disentanglement for Multimodal Sentiment Analysiscited by 5
  8. 2026CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learningcited by 5
  9. 2025TiCAL:Typicality-Based Consistency-Aware Learning for Multimodal Emotion Recognitioncited by 4
  10. 2025Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysiscited by 3
  11. 2025SAMSOD: Rethinking SAM Optimization for RGB-T Salient Object Detectioncited by 3
  12. 2026Temporal-Spatial Decouple before Act: Disentangled Representation Learning for Multimodal Sentiment Analysiscited by 3
22 more
  1. 2026Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaborationcited by 3
  2. 2025Multimodal Negative Learningcited by 2
  3. 2025Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learningcited by 2
  4. 2025Revisit Modality Imbalance at the Decision Layercited by 1
  5. 2026Balancing Multimodal Domain Generalization via Gradient Modulation and Projectioncited by 1
  6. 2026BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Ratescited by 1
  7. 2026PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritizationcited by 1
  8. 2026Does Text Actually Help? Uncovering and Resolving Text Collapse in Multimodal Time Series Forecastingcited by 1
  9. 2025CMD-HAR: Cross-Modal Disentanglement for Wearable Human Activity Recognitioncited by 0
  10. 2025A Robust Incomplete Multimodal Low-Rank Adaptation Approach for Emotion Recognitioncited by 0
  11. 2025Preserving Cross-Modal Stability for Visual Unlearning in Multimodal Scenarioscited by 0
  12. 2025MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulationcited by 0
  13. 2026Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptationcited by 0
  14. 2026CMKL: Modality-Aware Continual Learning for Evolving Biomedical Knowledge Graphscited by 0
  15. 2026IDO: Incongruity-aware Distribution Optimization for Multimodal Fake News Detectioncited by 0
  16. 2026ShapKO: Shapley-Adaptive Modality Knockout for Robust Multimodal Learningcited by 0
  17. 2026Balanced Soft mixture-of-expert model for Glaucoma Detectioncited by 0
  18. 2026UniMod: Enhancing Multi-Modal Medical Diagnosis through Cross-Modality and Within-Modality Alignmentcited by 0
  19. 2026The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methodscited by 0
  20. 2026SAGG: Sample-Adaptive Gradient Gating for Robust Multimodal Learning under Heterogeneous Corruptioncited by 0
  21. 2026Reassessing Global Gradient-Norm Imbalance in BLIP Fine-Tuning Across Physical Domainscited by 0
  22. 2026VCMM: Variance-Calibrated Momentum for Multimodal Learningcited by 0