mlsanityData to 14 Oct 2026

See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias

arXiv 2503.13834 ↗First posted 18 Mar 2025cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Vision-language (VL) models have demonstrated strong performance across various tasks. However, these models often rely on a specific modality for predictions, leading to "dominant modality bias.'' This bias significantly hurts performance, especially when one modality is impaired. In this study, we analyze model behavior under dominant modality bias and theoretically show that unaligned gradients or differences in gradient magnitudes prevent balanced convergence of the loss. Based on these findings, we propose a novel framework, BalGrad to mitigate dominant modality bias. Our approach includes inter-modality gradient reweighting, adjusting the gradient of KL divergence based on each modality's contribution, and inter-task gradient projection to align task directions in a non-conflicting manner. Experiments on UPMC Food-101, Hateful Memes, and MM-IMDb datasets confirm that BalGrad effectively alleviates over-reliance on specific modalities when making predictions.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

28 of 47 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  9. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  10. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
  11. 2017Gated Multimodal Units for Information Fusioncited by 248
  12. 2018Characterizing and Avoiding Negative Transfercited by 229
16 more
  1. 2018Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysiscited by 179
  2. 2021Towards Understanding Knowledge Distillationcited by 158
  3. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  4. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 122
  5. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  6. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 73
  7. 2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 58
  8. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 56
  9. 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 54
  10. 2019Integrating Text and Image: Determining Multimodal Document Intent in Instagram Postscited by 37
  11. 2021DATE: Detecting Anomalies in Text via Self-Supervision of Transformerscited by 25
  12. 2022On Modality Bias Recognition and Reductioncited by 20
  13. 2024Quantifying and Enhancing Multi-modal Robustness with Modality Preferencecited by 16
  14. 2023Memotion 3: Dataset on Sentiment and Emotion Analysis of Codemixed Hindi-English Memescited by 12
  15. 2022HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbationcited by 11
  16. 2024Text or Image? What is More Important in Cross-Domain Generalization Capabilities of Hate Meme Detection Models?cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Modelscited by 4
  2. 2026Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localizationcited by 1
  3. 2026Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusioncited by 1
  4. 2026Balancing Multimodal Domain Generalization via Gradient Modulation and Projectioncited by 1
  5. 2026VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendationcited by 0
  6. 2026Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activationcited by 0
  7. 2026Characterizing Text Branch Sensitivity in Medical Vision-Language Segmentation via Evidence Decouplingcited by 0
  8. 2026Reassessing Global Gradient-Norm Imbalance in BLIP Fine-Tuning Across Physical Domainscited by 0
  9. 2026To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speechcited by 0