# Improving Multimodal Learning via Imbalanced Learning

arXiv: https://arxiv.org/abs/2507.10203 · HTML: https://mlsanity.com/paper/2507.10203/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Builds on (most cited first, 25 of 25)

- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation](https://mlsanity.com/paper/1905.10089/) (2019)
- [mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation](https://mlsanity.com/paper/2206.02425/) (2022)
- [Multi-attention Recurrent Network for Human Communication Comprehension](https://mlsanity.com/paper/1802.00923/) (2018)
- [Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis](https://mlsanity.com/paper/2011.06961/) (2020)
- [Modality Distillation with Multiple Stream Networks for Action Recognition](https://mlsanity.com/paper/1806.07110/) (2018)
- [ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/2108.10528/) (2021)
- [Multimodal Representation Learning by Alternating Unimodal Adaptation](https://mlsanity.com/paper/2311.10707/) (2023)
- [Boosting Multi-modal Model Performance with Adaptive Gradient Modulation](https://mlsanity.com/paper/2308.07686/) (2023)
- [MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance](https://mlsanity.com/paper/2405.17730/) (2024)
- [ReconBoost: Boosting Can Achieve Modality Reconcilement](https://mlsanity.com/paper/2405.09321/) (2024)
- [Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion](https://mlsanity.com/paper/2103.11832/) (2021)
- [Multimodal Fusion on Low-quality Data: A Comprehensive Survey](https://mlsanity.com/paper/2404.18947/) (2024)
- [Improving Multi-Modal Learning with Uni-Modal Teachers](https://mlsanity.com/paper/2106.11059/) (2021)
- [Diagnosing and Re-learning for Balanced Multimodal Learning](https://mlsanity.com/paper/2407.09705/) (2024)
- [MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning](https://mlsanity.com/paper/2304.08028/) (2023)
- [MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning](https://mlsanity.com/paper/2303.05338/) (2023)
- [Robust Multimodal Learning via Representation Decoupling](https://mlsanity.com/paper/2407.04458/) (2024)
- [Optimizing Ensemble Weights and Hyperparameters of Machine Learning Models for Regression Problems](https://mlsanity.com/paper/1908.05287/) (2019)
- [Learnable Irrelevant Modality Dropout for Multimodal Action Recognition on Modality-Specific Annotated Videos](https://mlsanity.com/paper/2203.03014/) (2022)

## Built on by (most cited first, 17 of 17)

- [Unbiased Dynamic Multimodal Fusion](https://mlsanity.com/paper/2603.19681/) (2026)
- [Fusion or Confusion? Multimodal Complexity Is Not All You Need](https://mlsanity.com/paper/2512.22991/) (2025)
- [CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning](https://mlsanity.com/paper/2602.19605/) (2026)
- [Visual Enhanced Depth Scaling for Multimodal Latent Reasoning](https://mlsanity.com/paper/2604.10500/) (2026)
- [Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusion](https://mlsanity.com/paper/2601.21670/) (2026)
- [PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization](https://mlsanity.com/paper/2604.05773/) (2026)
- [DRFormer: A Dual-Regularized Bidirectional Transformer for Person Re-identification](https://mlsanity.com/paper/2602.01059/) (2026)
- [Parameter-Efficient Modality-Balanced Symmetric Fusion for Multimodal Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2603.17705/) (2026)
- [Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration](https://mlsanity.com/paper/2605.03820/) (2026)
- [Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals](https://mlsanity.com/paper/2606.02679/) (2026)
- [General Incomplete Multimodal Learning via Dynamic Quality Perception](https://mlsanity.com/paper/2607.06943/) (2026)
- [Beyond Symmetric Fusion: Exploiting Task-Dependent Modality Strengths for RGB-Event Small Object Detection](https://mlsanity.com/paper/2608.01302/) (2026)
- [SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding](https://mlsanity.com/paper/2608.05137/) (2026)
- [SuppreSensing: Expert-Guided Feature Recalibration and Discrepancy Augmentation for Multimodal Object Detection](https://mlsanity.com/paper/2608.20944/) (2026)
- [Contribution-Aware Bandwidth Allocation for Multimodal Split Learning](https://mlsanity.com/paper/2609.01406/) (2026)
- [Reassessing Global Gradient-Norm Imbalance in BLIP Fine-Tuning Across Physical Domains](https://mlsanity.com/paper/2609.23655/) (2026)
- [Balancing Multimodal Learning via Functional Progress](https://mlsanity.com/paper/2610.03035/) (2026)

## Abstract (arXiv)

Multimodal learning often encounters the under-optimized problem and may perform worse than unimodal learning. Existing approaches attribute this issue to imbalanced learning across modalities and tend to address it through gradient balancing. However, this paper argues that balanced learning is not the optimal setting for multimodal learning. With bias-variance analysis, we prove that imbalanced dependency on each modality obeying the inverse ratio of their variances contributes to optimal performance. To this end, we propose the Asymmetric Representation Learning(ARL) strategy to assist multimodal learning via imbalanced optimization. ARL introduces auxiliary regularizers for each modality encoder to calculate their prediction variance. ARL then calculates coefficients via the unimodal variance to re-weight the optimization of each modality, forcing the modality dependence ratio to be inversely proportional to the modality variance ratio. Moreover, to minimize the generalization error, ARL further introduces the prediction bias of each modality and jointly optimizes them with multimodal loss. Notably, all auxiliary regularizers share parameters with the multimodal model and rely only on the modality representation. Thus the proposed ARL strategy introduces no extra parameters and is independent of the structures and fusion methods of the multimodal model. Finally, extensive experiments on various datasets validate the effectiveness and versatility of ARL. Code is available at hrefhttps://github.com/shicaiwei123/ICCV2025-ARLhttps://github.com/shicaiwei123/ICCV2025-ARL