# Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition

arXiv: https://arxiv.org/abs/2503.18595 · HTML: https://mlsanity.com/paper/2503.18595/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [The large learning rate phase of deep learning: the catapult mechanism](https://mlsanity.com/paper/2003.02218/) (2020)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [The Break-Even Point on Optimization Trajectories of Deep Neural Networks](https://mlsanity.com/paper/2002.09572/) (2020)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Scaling Laws for Generative Mixed-Modal Language Models](https://mlsanity.com/paper/2301.03728/) (2023)
- [Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)](https://mlsanity.com/paper/2203.12221/) (2022)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length](https://mlsanity.com/paper/1807.05031/) (2018)
- [Boosting Multi-modal Model Performance with Adaptive Gradient Modulation](https://mlsanity.com/paper/2308.07686/) (2023)
- [Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization](https://mlsanity.com/paper/2012.14193/) (2020)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence](https://mlsanity.com/paper/1905.13277/) (2019)
- [On the interplay between noise and curvature and its effect on optimization and generalization](https://mlsanity.com/paper/1906.07774/) (2019)
- [Improving Multi-Modal Learning with Uni-Modal Teachers](https://mlsanity.com/paper/2106.11059/) (2021)

## Built on by (most cited first, 19 of 19)

- [Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalization](https://mlsanity.com/paper/2511.20258/) (2025)
- [Multimodal Negative Learning](https://mlsanity.com/paper/2510.20877/) (2025)
- [RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer](https://mlsanity.com/paper/2506.11465/) (2025)
- [Crab+: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation](https://mlsanity.com/paper/2603.04128/) (2026)
- [Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities](https://mlsanity.com/paper/2603.09111/) (2026)
- [Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2604.12518/) (2026)
- [Reliable Microservice Tail Latency Prediction via Decoupled Dual-Stream Learning and Gradient Modulation](https://mlsanity.com/paper/2508.01635/) (2025)
- [Plug, Play, and Fortify: A Low-Cost Module for Robust Multimodal Image Understanding Models](https://mlsanity.com/paper/2602.22644/) (2026)
- [When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion](https://mlsanity.com/paper/2602.23614/) (2026)
- [DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning](https://mlsanity.com/paper/2603.23916/) (2026)
- [Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration](https://mlsanity.com/paper/2605.03820/) (2026)
- [Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2605.21565/) (2026)
- [Uncertainty-aware damage identification in short-span bridges via physics-informed variational autoencoder](https://mlsanity.com/paper/2607.05025/) (2026)
- [Understanding and Overcoming Cross-modal Fusion Bias in Multimodal Anomaly Detection From A Fisher Information Perspective](https://mlsanity.com/paper/2608.00986/) (2026)
- [If You Hear It, Help Find It: Orthogonal Knowledge Distillation for Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2609.23376/) (2026)
- [GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learning](https://mlsanity.com/paper/2609.23533/) (2026)
- [VCMM: Variance-Calibrated Momentum for Multimodal Learning](https://mlsanity.com/paper/2609.27577/) (2026)
- [Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification](https://mlsanity.com/paper/2609.28165/) (2026)
- [Balancing Multimodal Learning via Functional Progress](https://mlsanity.com/paper/2610.03035/) (2026)

## Abstract (arXiv)

Sensory training during the early ages is vital for human development. Inspired by this cognitive phenomenon, we observe that the early training stage is also important for the multimodal learning process, where dataset information is rapidly acquired. We refer to this stage as the prime learning window. However, based on our observation, this prime learning window in multimodal learning is often dominated by information-sufficient modalities, which in turn suppresses the information acquisition of information-insufficient modalities. To address this issue, we propose Information Acquisition Regulation (InfoReg), a method designed to balance information acquisition among modalities. Specifically, InfoReg slows down the information acquisition process of information-sufficient modalities during the prime learning window, which could promote information acquisition of information-insufficient modalities. This regulation enables a more balanced learning process and improves the overall performance of the multimodal network. Experiments show that InfoReg outperforms related multimodal imbalanced methods across various datasets, achieving superior model performance. The code is available at https://github.com/GeWu-Lab/InfoReg_CVPR2025.