# DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning

arXiv: https://arxiv.org/abs/2503.06456 · HTML: https://mlsanity.com/paper/2503.06456/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 9 Mar 2025.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [EMMA: End-to-End Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2410.23262/) (2024)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Trusted Multi-View Classification with Dynamic Evidential Fusion](https://mlsanity.com/paper/2204.11423/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2412.15208/) (2024)
- [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://mlsanity.com/paper/2306.02050/) (2023)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [Decoupled Multimodal Distilling for Emotion Recognition](https://mlsanity.com/paper/2303.13802/) (2023)
- [Multimodal Representation Learning by Alternating Unimodal Adaptation](https://mlsanity.com/paper/2311.10707/) (2023)
- [Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement](https://mlsanity.com/paper/2405.15973/) (2024)
- [MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance](https://mlsanity.com/paper/2405.17730/) (2024)
- [ReconBoost: Boosting Can Achieve Modality Reconcilement](https://mlsanity.com/paper/2405.09321/) (2024)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Multimodal Fusion on Low-quality Data: A Comprehensive Survey](https://mlsanity.com/paper/2404.18947/) (2024)
- [Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2312.06337/) (2023)
- [Enhancing multimodal cooperation via sample-level modality valuation](https://mlsanity.com/paper/2309.06255/) (2023)
- [Improving Multi-Modal Learning with Uni-Modal Teachers](https://mlsanity.com/paper/2106.11059/) (2021)
- [Classifier-guided Gradient Modulation for Enhanced Multimodal Learning](https://mlsanity.com/paper/2411.01409/) (2024)
- [AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving](https://mlsanity.com/paper/2412.15206/) (2024)
- [On-the-fly Modulation for Balanced Multimodal Learning](https://mlsanity.com/paper/2410.11582/) (2024)
- [Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization](https://mlsanity.com/paper/2502.13146/) (2025)
- [Predictive Dynamic Fusion](https://mlsanity.com/paper/2406.04802/) (2024)

## Built on by (most cited first, 16 of 16)

- [MAT-Agent: Adaptive Multi-Agent Training Optimization](https://mlsanity.com/paper/2510.17845/) (2025)
- [HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval](https://mlsanity.com/paper/2512.02792/) (2025)
- [Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools](https://mlsanity.com/paper/2510.08480/) (2025)
- [DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learning](https://mlsanity.com/paper/2503.11892/) (2025)
- [PROGRESSLM: Towards Progress Reasoning in Vision-Language Models](https://mlsanity.com/paper/2601.15224/) (2026)
- [DVP-MVS++: Synergize Depth-Normal-Edge and Harmonized Visibility Prior for Multi-View Stereo](https://mlsanity.com/paper/2506.13215/) (2025)
- [CLIMD: A Curriculum Learning Framework for Imbalanced Multimodal Diagnosis](https://mlsanity.com/paper/2508.01594/) (2025)
- [End-to-End Multi-Modal Diffusion Mamba](https://mlsanity.com/paper/2510.13253/) (2025)
- [MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning](https://mlsanity.com/paper/2509.25831/) (2025)
- [Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models](https://mlsanity.com/paper/2605.16165/) (2026)
- [Mixed Signals: Understanding Model Disagreement in Multimodal Empathy Detection](https://mlsanity.com/paper/2505.13979/) (2025)
- [LiMT: A Multi-task Liver Image Benchmark Dataset](https://mlsanity.com/paper/2511.19889/) (2025)
- [HarmTransform: Transforming Explicit Harmful Queries into Stealthy via Multi-Agent Debate](https://mlsanity.com/paper/2512.23717/) (2025)
- [DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation](https://mlsanity.com/paper/2506.11691/) (2025)
- [Multi-Modal Sentiment Analysis with Dynamic Attention Fusion](https://mlsanity.com/paper/2509.22729/) (2025)
- [Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2605.21565/) (2026)

## Abstract (arXiv)

Multimodal learning integrates complementary information from diverse modalities to enhance the decision-making process. However, the potential of multimodal collaboration remains under-exploited due to disparities in data quality and modality representation capabilities. To address this, we introduce DynCIM, a novel dynamic curriculum learning framework designed to quantify the inherent imbalances from both sample and modality perspectives. DynCIM employs a sample-level curriculum to dynamically assess each sample's difficulty according to prediction deviation, consistency, and stability, while a modality-level curriculum measures modality contributions from global and local. Furthermore, a gating-based dynamic fusion mechanism is introduced to adaptively adjust modality contributions, minimizing redundancy and optimizing fusion effectiveness. Extensive experiments on six multimodal benchmarking datasets, spanning both bimodal and trimodal scenarios, demonstrate that DynCIM consistently outperforms state-of-the-art methods. Our approach effectively mitigates modality and sample imbalances while enhancing adaptability and robustness in multimodal learning tasks. Our code is available at https://github.com/Raymond-Qiancx/DynCIM.