# Classifier-guided Gradient Modulation for Enhanced Multimodal Learning

arXiv: https://arxiv.org/abs/2411.01409 · HTML: https://mlsanity.com/paper/2411.01409/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 3 Nov 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [MUTAN: Multimodal Tucker Fusion for Visual Question Answering](https://mlsanity.com/paper/1705.06676/) (2017)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)](https://mlsanity.com/paper/2203.12221/) (2022)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors](https://mlsanity.com/paper/1811.09362/) (2018)
- [CentralNet: a Multilayer Approach for Multimodal Fusion](https://mlsanity.com/paper/1808.07275/) (2018)
- [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://mlsanity.com/paper/2306.02050/) (2023)
- [Boosting Multi-modal Model Performance with Adaptive Gradient Modulation](https://mlsanity.com/paper/2308.07686/) (2023)
- [ReconBoost: Boosting Can Achieve Modality Reconcilement](https://mlsanity.com/paper/2405.09321/) (2024)
- [On Uni-Modal Feature Learning in Supervised Multi-Modal Learning](https://mlsanity.com/paper/2305.01233/) (2023)
- [Multimodal Language Analysis with Recurrent Multistage Fusion](https://mlsanity.com/paper/1808.03920/) (2018)
- [Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2407.05374/) (2024)
- [Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2307.02041/) (2023)

## Built on by (most cited first, 30 of 35)

- [DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learning](https://mlsanity.com/paper/2503.11892/) (2025)
- [DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning](https://mlsanity.com/paper/2503.06456/) (2025)
- [Towards Balanced Multi-Modal Learning in 3D Human Pose Estimation](https://mlsanity.com/paper/2501.05264/) (2025)
- [Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises](https://mlsanity.com/paper/2503.02616/) (2025)
- [Efficient Prompting for Continual Adaptation to Missing Modalities](https://mlsanity.com/paper/2503.00528/) (2025)
- [Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalization](https://mlsanity.com/paper/2511.20258/) (2025)
- [Tri-Subspaces Disentanglement for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2602.19585/) (2026)
- [TiCAL:Typicality-Based Consistency-Aware Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2511.15085/) (2025)
- [CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning](https://mlsanity.com/paper/2602.19605/) (2026)
- [Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysis](https://mlsanity.com/paper/2504.12151/) (2025)
- [SAMSOD: Rethinking SAM Optimization for RGB-T Salient Object Detection](https://mlsanity.com/paper/2510.03689/) (2025)
- [Multimodal Negative Learning](https://mlsanity.com/paper/2510.20877/) (2025)
- [Temporal-Spatial Decouple before Act: Disentangled Representation Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2601.13659/) (2026)
- [Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaboration](https://mlsanity.com/paper/2605.00370/) (2026)
- [Does Text Actually Help? Uncovering and Resolving Text Collapse in Multimodal Time Series Forecasting](https://mlsanity.com/paper/2606.19413/) (2026)
- [Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning](https://mlsanity.com/paper/2510.24919/) (2025)
- [Revisit Modality Imbalance at the Decision Layer](https://mlsanity.com/paper/2510.14411/) (2025)
- [Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptation](https://mlsanity.com/paper/2603.00574/) (2026)
- [Balancing Multimodal Domain Generalization via Gradient Modulation and Projection](https://mlsanity.com/paper/2603.14175/) (2026)
- [BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates](https://mlsanity.com/paper/2603.19718/) (2026)
- [PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization](https://mlsanity.com/paper/2604.05773/) (2026)
- [CMD-HAR: Cross-Modal Disentanglement for Wearable Human Activity Recognition](https://mlsanity.com/paper/2503.21843/) (2025)
- [A Robust Incomplete Multimodal Low-Rank Adaptation Approach for Emotion Recognition](https://mlsanity.com/paper/2507.11202/) (2025)
- [Preserving Cross-Modal Stability for Visual Unlearning in Multimodal Scenarios](https://mlsanity.com/paper/2509.23895/) (2025)
- [MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulation](https://mlsanity.com/paper/2510.07328/) (2025)
- [CMKL: Modality-Aware Continual Learning for Evolving Biomedical Knowledge Graphs](https://mlsanity.com/paper/2605.10510/) (2026)
- [Balancing Multimodal Learning through Label Space Reshaping](https://mlsanity.com/paper/2605.28869/) (2026)
- [IDO: Incongruity-aware Distribution Optimization for Multimodal Fake News Detection](https://mlsanity.com/paper/2606.03418/) (2026)
- [ShapKO: Shapley-Adaptive Modality Knockout for Robust Multimodal Learning](https://mlsanity.com/paper/2607.09884/) (2026)
- [Balanced Soft mixture-of-expert model for Glaucoma Detection](https://mlsanity.com/paper/2607.25324/) (2026)

## Abstract (arXiv)

Multimodal learning has developed very fast in recent years. However, during the multimodal training process, the model tends to rely on only one modality based on which it could learn faster, thus leading to inadequate use of other modalities. Existing methods to balance the training process always have some limitations on the loss functions, optimizers and the number of modalities and only consider modulating the magnitude of the gradients while ignoring the directions of the gradients. To solve these problems, in this paper, we present a novel method to balance multimodal learning with Classifier-Guided Gradient Modulation (CGGM), considering both the magnitude and directions of the gradients. We conduct extensive experiments on four multimodal datasets: UPMC-Food 101, CMU-MOSI, IEMOCAP and BraTS 2021, covering classification, regression and segmentation tasks. The results show that CGGM outperforms all the baselines and other state-of-the-art methods consistently, demonstrating its effectiveness and versatility. Our code is available at https://github.com/zrguo/CGGM.