mlsanityData to 14 Oct 2026

MLLMs are Deeply Affected by Modality Bias

arXiv 2505.18657 ↗First posted 24 May 2025cs.AIcited by 40

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent advances in Multimodal Large Language Models (MLLMs) have shown promising results in integrating diverse modalities such as texts and images. MLLMs are heavily influenced by modality bias, often relying on language while under-utilizing other modalities like visual inputs. This position paper argues that MLLMs are deeply affected by modality bias. Firstly, we diagnose the current state of modality bias, highlighting its manifestations across various tasks. Secondly, we propose a systematic research road-map related to modality bias in MLLMs. Thirdly, we identify key factors of modality bias in MLLMs and offer actionable suggestions for future research to mitigate it. To substantiate these findings, we conduct experiments that demonstrate the influence of each factor: 1. Data Characteristics: Language data is compact and abstract, while visual data is redundant and complex, creating an inherent imbalance in learning dynamics. 2. Imbalanced Backbone Capabilities: The dominance of pretrained language models in MLLMs leads to overreliance on language and neglect of visual information. 3. Training Objectives: Current objectives often fail to promote balanced cross-modal alignment, resulting in shortcut learning biased toward language. These findings highlight the need for balanced training strategies and model architectures to better integrate multiple modalities in MLLMs. We call for interdisciplinary efforts to tackle these challenges and drive innovation in MLLM research. Our work provides a fresh perspective on modality bias in MLLMs and offers insights for developing more robust and generalizable multimodal systems-advancing progress toward Artificial General Intelligence.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 89 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

74 of 89 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization23 Mar
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance21 Nov
  4. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs10 Oct
  5. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models21 Sep
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  7. 2024Visual Prompting in Multimodal Large Language Models: A Survey5 Sep
  8. 2024Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models22 Aug
  9. 2024A Survey on Benchmarks of Multimodal Large Language Models16 Aug
  10. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective27 Mar
  11. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization13 Mar
  12. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priors8 Mar
4 more
  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  2. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  4. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  8. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  10. 2019Event-based Vision: A Surveycited by 1,016
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  12. 2020A Survey on Visual Transformercited by 654
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 648
  2. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  3. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  4. 2022Multimodal Learning with Transformers: A Surveycited by 400
  5. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  6. 2019High Speed and High Dynamic Range Video with an Event Cameracited by 345
  7. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  8. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  9. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  10. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  11. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  12. 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 105
  13. 2023Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarkscited by 90
  14. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  15. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  16. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  17. 2020Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropiescited by 57
  18. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 55
  19. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
  20. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  21. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 45
  22. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 44
  23. 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
  24. 2024Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentationcited by 34
  25. 2022Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformerscited by 33
  26. 2024CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenescited by 33
  27. 2024Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Trainingcited by 31
  28. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalitiescited by 31
  29. 2022A Review on Methods and Applications in Multimodal Deep Learningcited by 29
  30. 2024360SFUDA++: Towards Source-free UDA for Panoramic Segmentation by Learning Reliable Category Prototypescited by 28
  31. 2025OmniSAM: Omnidirectional Segment Anything Model for UDA in Panoramic Semantic Segmentationcited by 28
  32. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  33. 2023Learning Disentangled Identifiers for Action-Customized Text-to-Image Generationcited by 24
  34. 2024Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentationcited by 22
  35. 2024Bi-VLA: Vision-Language-Action Model-Based System for Bimanual Robotic Dexterous Manipulationscited by 21
  36. 2022On Modality Bias Recognition and Reductioncited by 20
  37. 2024OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of Allcited by 20
  38. 2025Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustnesscited by 18

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 28.

40 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025MemER: Scaling Up Memory for Robot Control via Experience Retrievalcited by 67
  3. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  4. 2026When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAscited by 30
  5. 2025When Language Overrules: Revealing Text Dominance in Multimodal Large Language Modelscited by 18
  6. 2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 9
  7. 2026Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstentioncited by 6
  8. 2025Omnidirectional Spatial Modeling from Correlated Panoramascited by 5
  9. 2026When Audio-LLMs Don't Listen: A Cross-Linguistic Study of Modality Arbitrationcited by 5
  10. 2026Vesta: A Generalist Embodied Reasoning Modelcited by 5
  11. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  12. 2026Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steeringcited by 4
28 more
  1. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judgecited by 4
  2. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Abilitycited by 3
  3. 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Biascited by 3
  4. 2026Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integrationcited by 3
  5. 2025FinCap: Topic-Aligned Captions for Short-Form Financial YouTube Videoscited by 2
  6. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  7. 2025Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?cited by 2
  8. 2026MIBench: Evaluating LMMs on Multimodal Interactioncited by 2
  9. 2026Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMscited by 2
  10. 2026CAAD: Contrastive Audio-Aware Distillation for Efficient Speech Language Modelscited by 2
  11. 2026C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Modelscited by 2
  12. 2026Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localizationcited by 1
  13. 2026Seeing Beyond 8bits: Subjective and Objective Quality Assessment of HDR-UGC Videoscited by 1
  14. 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 1
  15. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainabilitycited by 1
  16. 2026Omni-Streaming Thinkingcited by 1
  17. 2025CogGuide: Human-Like Guidance for Zero-Shot Omni-Modal Reasoningcited by 0
  18. 2025Understanding Temporal Logic Consistency in Video-Language Models through Cross-Modal Attention Discriminabilitycited by 0
  19. 2026Token-Efficient Multimodal Reasoning via Image Prompt Packagingcited by 0
  20. 2026A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoningcited by 0
  21. 2026Simulating Validity: Modal Decoupling in MLLM Generated Feedback on Science Drawingscited by 0
  22. 2026Do Composed Image Retrieval Benchmarks Require Multimodal Composition?cited by 0
  23. 2026Semantic Generative Tuning for Unified Multimodal Modelscited by 0
  24. 2026EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memorycited by 0
  25. 2026Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflictscited by 0
  26. 2026TryOnReward: Learning Foveated Consistency for Reinforcement Fine-Tuning of Virtual Try-Oncited by 0
  27. 2026GraphWrit3R: End-to-End 3D Scene Graph Writingcited by 0
  28. 2026Scaling Laws for Multimodal Data Mixturescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes
  2. 2025Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
  3. 2026Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
  4. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  5. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly
  6. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
  7. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models
  8. 2024The Instinctive Bias: Spurious Images lead to Illusion in MLLMs