mlsanityData to 7 Oct 2026

Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition

arXiv 2503.18595 ↗First posted 24 Mar 2025cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Sensory training during the early ages is vital for human development. Inspired by this cognitive phenomenon, we observe that the early training stage is also important for the multimodal learning process, where dataset information is rapidly acquired. We refer to this stage as the prime learning window. However, based on our observation, this prime learning window in multimodal learning is often dominated by information-sufficient modalities, which in turn suppresses the information acquisition of information-insufficient modalities. To address this issue, we propose Information Acquisition Regulation (InfoReg), a method designed to balance information acquisition among modalities. Specifically, InfoReg slows down the information acquisition process of information-sufficient modalities during the prime learning window, which could promote information acquisition of information-insufficient modalities. This regulation enables a more balanced learning process and improves the overall performance of the multimodal network. Experiments show that InfoReg outperforms related multimodal imbalanced methods across various datasets, achieving superior model performance. The code is available at https://github.com/GeWu-Lab/InfoReg_CVPR2025.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

34 of 49 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  6. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  7. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  8. 2017Look, Listen and Learncited by 452
  9. 2014New insights and perspectives on the natural gradient methodcited by 435
  10. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  11. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  12. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
22 more
  1. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  2. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  3. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  4. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  5. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  6. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  7. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  8. 2020The Early Phase of Neural Network Trainingcited by 114
  9. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 114
  10. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  11. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  12. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  13. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 71
  14. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 61
  15. 2018Efficient Large-Scale Multi-Modal Classificationcited by 57
  16. 2019Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergencecited by 54
  17. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  18. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 40
  19. 2024Diagnosing and Re-learning for Balanced Multimodal Learningcited by 40
  20. 2020Temporal Relational Modeling with Self-Supervision for Action Segmentationcited by 29
  21. 2020Improving Multimodal Accuracy Through Modality Pre-training and Attentioncited by 15
  22. 2021Critical Learning Periods in Federated Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 14.

18 papers.

Sort:

  1. 2025Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalizationcited by 6
  2. 2025Multimodal Negative Learningcited by 2
  3. 2025RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformercited by 1
  4. 2026Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 1
  5. 2026Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysiscited by 1
  6. 2025Reliable Microservice Tail Latency Prediction via Decoupled Dual-Stream Learning and Gradient Modulationcited by 0
  7. 2026Plug, Play, and Fortify: A Low-Cost Module for Robust Multimodal Image Understanding Modelscited by 0
  8. 2026When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusioncited by 0
  9. 2026Crab+: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 0
  10. 2026DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learningcited by 0
  11. 2026Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibrationcited by 0
  12. 2026Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognitioncited by 0
6 more
  1. 2026Uncertainty-aware damage identification in short-span bridges via physics-informed variational autoencodercited by 0
  2. 2026Understanding and Overcoming Cross-modal Fusion Bias in Multimodal Anomaly Detection From A Fisher Information Perspectivecited by 0
  3. 2026If You Hear It, Help Find It: Orthogonal Knowledge Distillation for Open-Vocabulary Audio-Visual Event Localizationcited by 0
  4. 2026GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learningcited by 0
  5. 2026VCMM: Variance-Calibrated Momentum for Multimodal Learningcited by 0
  6. 2026Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classificationcited by 0