# Balancing Multimodal Domain Generalization via Gradient Modulation and Projection

arXiv: https://arxiv.org/abs/2603.14175 · HTML: https://mlsanity.com/paper/2603.14175/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 15 Mar 2026.

## Builds on (most cited first, 21 of 21)

- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Boosting Multi-modal Model Performance with Adaptive Gradient Modulation](https://mlsanity.com/paper/2308.07686/) (2023)
- [SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization](https://mlsanity.com/paper/2310.19795/) (2023)
- [Diagnosing and Re-learning for Balanced Multimodal Learning](https://mlsanity.com/paper/2407.09705/) (2024)
- [Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognition](https://mlsanity.com/paper/2110.10101/) (2021)
- [Classifier-guided Gradient Modulation for Enhanced Multimodal Learning](https://mlsanity.com/paper/2411.01409/) (2024)
- [M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering](https://mlsanity.com/paper/2504.04633/) (2025)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration](https://mlsanity.com/paper/2505.17098/) (2025)
- [Towards Multimodal Open-Set Domain Generalization and Adaptation through Self-supervision](https://mlsanity.com/paper/2407.01518/) (2024)
- [See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias](https://mlsanity.com/paper/2503.13834/) (2025)
- [Domain Adaptive Object Detection via Balancing Between Self-Training and Adversarial Learning](https://mlsanity.com/paper/2311.04815/) (2023)
- [Improving Pseudo-labelling and Enhancing Robustness for Semi-Supervised Domain Generalization](https://mlsanity.com/paper/2401.13965/) (2024)
- [TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability](https://mlsanity.com/paper/2405.17678/) (2024)
- [Towards Multimodal Domain Generalization with Few Labels](https://mlsanity.com/paper/2602.22917/) (2026)

## Built on by (most cited first, 1 of 1)

- [Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study](https://mlsanity.com/paper/2605.06643/) (2026)

## Abstract (arXiv)

Multimodal Domain Generalization (MMDG) leverages the complementary strengths of multiple modalities to enhance model generalization on unseen domains. A central challenge in multimodal learning is optimization imbalance, where modalities converge at different speeds during training. This imbalance leads to unequal gradient contributions, allowing some modalities to dominate the learning process while others lag behind. Existing balancing strategies typically regulate each modality's gradient contribution based on its classification performance on the source domain to alleviate this issue. However, relying solely on source-domain accuracy neglects a key insight in MMDG: modalities that excel on the source domain may generalize poorly to unseen domains, limiting cross-domain gains. To overcome this limitation, we propose Gradient Modulation Projection (GMP), a unified strategy that promotes balanced optimization in MMDG. GMP first decouples gradients associated with classification and domain-invariance objectives. It then modulates each modality's gradient based on semantic and domain confidence. Moreover, GMP dynamically adjusts gradient projections by tracking the relative strength of each task, mitigating conflicts between classification and domain-invariant learning within modality-specific encoders. Extensive experiments demonstrate that GMP achieves state-of-the-art performance and integrates flexibly with diverse MMDG methods, significantly improving generalization across multiple benchmarks.