mlsanityData to 14 Oct 2026

When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models

arXiv 2508.10552 ↗First posted 14 Aug 2025cs.CL cs.AIcited by 18

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across a diverse range of multimodal tasks. However, these models suffer from a core problem known as text dominance: they depend heavily on text for their inference, while underutilizing other modalities. While prior work has acknowledged this phenomenon in vision-language tasks, often attributing it to data biases or model architectures. In this paper, we conduct the first systematic investigation of text dominance across diverse data modalities, including images, videos, audio, time-series, and graphs. To measure this imbalance, we propose two evaluation metrics: the Modality Dominance Index (MDI) and the Attention Efficiency Index (AEI). Our comprehensive analysis reveals that text dominance is both significant and pervasive across all tested modalities. Our in-depth analysis identifies three underlying causes: attention dilution from severe token redundancy in non-textual modalities, the influence of fusion architecture design, and task formulations that implicitly favor textual inputs. Furthermore, we propose a simple token compression method that effectively rebalances model attention. Applying this method to LLaVA-7B, for instance, drastically reduces its MDI from 10.23 to a well-balanced value of 0.86. Our analysis and methodological framework offer a foundation for the development of more equitable and comprehensive multimodal language models.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 19 references that are in the map. First posted on arXiv 14 Aug 2025.

Builds on

13 of 19 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MLLMs are Deeply Affected by Modality Bias24 May
  2. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining20 May
  3. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization17 Nov
  4. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  5. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training10 Oct
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 7

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  2. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  3. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  4. 2024ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoningcited by 103
  5. 2024MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrievalcited by 9
  6. 2025Improving Multimodal Learning Balance and Sufficiency through Data Remixingcited by 8
  7. 2024Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Interventioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 15.

18 papers.

Sort:

By area:

  1. 2026Watch Before You Answer: Learning from Visually Grounded Post-Trainingcited by 5
  2. 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoningcited by 4
  3. 2025MaLoRA: Gated Modality LoRA for Key-Space Alignment in Multimodal LLM Fine-Tuningcited by 4
  4. 2026MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understandingcited by 4
  5. 2026Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucinationcited by 4
  6. 2026SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognitioncited by 3
  7. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  8. 2026Information Router for Mitigating Modality Dominance in Vision-Language Modelscited by 1
  9. 2026Allocation Before Ranking: Decoupled Token Compression for OmniLLMscited by 1
  10. 2026Token-Efficient Multimodal Reasoning via Image Prompt Packagingcited by 0
  11. 2026MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignmentcited by 0
  12. 2026Attention is Case-Sensitivecited by 0
6 more
  1. 2026Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activationcited by 0
  2. 2026When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMscited by 0
  3. 2026Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Modelscited by 0
  4. 2026Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflictscited by 0
  5. 2026Multimodal Safety Evaluation Should Measure Controllability Beyond Classificationcited by 0
  6. 2026Scaling Laws for Multimodal Data Mixturescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SAISA: Towards Multimodal Large Language Models with Both Training and Inference Efficiency
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs
  4. 2026Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction
  5. 2025Multimodal Language Models See Better When They Look Shallower
  6. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models
  7. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks