# When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2508.10552 · HTML: https://mlsanity.com/paper/2508.10552/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 13 of 13)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization](https://mlsanity.com/paper/2411.10442/) (2024)
- [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding](https://mlsanity.com/paper/2406.14515/) (2024)
- [ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning](https://mlsanity.com/paper/2412.03104/) (2024)
- [Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training](https://mlsanity.com/paper/2410.08202/) (2024)
- [MLLMs are Deeply Affected by Modality Bias](https://mlsanity.com/paper/2505.18657/) (2025)
- [SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization](https://mlsanity.com/paper/2411.11909/) (2024)
- [MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval](https://mlsanity.com/paper/2406.17880/) (2024)
- [Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining](https://mlsanity.com/paper/2506.08022/) (2025)
- [Improving Multimodal Learning Balance and Sufficiency through Data Remixing](https://mlsanity.com/paper/2506.11550/) (2025)
- [Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention](https://mlsanity.com/paper/2410.10184/) (2024)

## Built on by (most cited first, 18 of 18)

- [Watch Before You Answer: Learning from Visually Grounded Post-Training](https://mlsanity.com/paper/2604.05117/) (2026)
- [Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning](https://mlsanity.com/paper/2509.23744/) (2025)
- [MaLoRA: Gated Modality LoRA for Key-Space Alignment in Multimodal LLM Fine-Tuning](https://mlsanity.com/paper/2510.26721/) (2025)
- [MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understanding](https://mlsanity.com/paper/2604.00513/) (2026)
- [Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination](https://mlsanity.com/paper/2608.07302/) (2026)
- [SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognition](https://mlsanity.com/paper/2603.17729/) (2026)
- [Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models](https://mlsanity.com/paper/2512.06281/) (2025)
- [Information Router for Mitigating Modality Dominance in Vision-Language Models](https://mlsanity.com/paper/2604.16264/) (2026)
- [Allocation Before Ranking: Decoupled Token Compression for OmniLLMs](https://mlsanity.com/paper/2608.01665/) (2026)
- [Token-Efficient Multimodal Reasoning via Image Prompt Packaging](https://mlsanity.com/paper/2604.02492/) (2026)
- [MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment](https://mlsanity.com/paper/2604.21326/) (2026)
- [Attention is Case-Sensitive](https://mlsanity.com/paper/2608.03711/) (2026)
- [Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activation](https://mlsanity.com/paper/2608.14655/) (2026)
- [When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs](https://mlsanity.com/paper/2609.04281/) (2026)
- [Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models](https://mlsanity.com/paper/2609.05401/) (2026)
- [Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts](https://mlsanity.com/paper/2609.06011/) (2026)
- [Multimodal Safety Evaluation Should Measure Controllability Beyond Classification](https://mlsanity.com/paper/2610.06452/) (2026)
- [Scaling Laws for Multimodal Data Mixtures](https://mlsanity.com/paper/2610.14423/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across a diverse range of multimodal tasks. However, these models suffer from a core problem known as text dominance: they depend heavily on text for their inference, while underutilizing other modalities. While prior work has acknowledged this phenomenon in vision-language tasks, often attributing it to data biases or model architectures. In this paper, we conduct the first systematic investigation of text dominance across diverse data modalities, including images, videos, audio, time-series, and graphs. To measure this imbalance, we propose two evaluation metrics: the Modality Dominance Index (MDI) and the Attention Efficiency Index (AEI). Our comprehensive analysis reveals that text dominance is both significant and pervasive across all tested modalities. Our in-depth analysis identifies three underlying causes: attention dilution from severe token redundancy in non-textual modalities, the influence of fusion architecture design, and task formulations that implicitly favor textual inputs. Furthermore, we propose a simple token compression method that effectively rebalances model attention. Applying this method to LLaVA-7B, for instance, drastically reduces its MDI from 10.23 to a well-balanced value of 0.86. Our analysis and methodological framework offer a foundation for the development of more equitable and comprehensive multimodal language models.