Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
Not on the map. Why?
Abstract (arXiv)
Native Omni-modal Large Language Models (OLLMs) have shifted from pipeline architectures to unified representation spaces. However, this native integration gives rise to a critical yet underexplored phenomenon: modality preference. To bridge this gap, we first systematically quantify modality preference of OLLMs using a newly-curated conflict-based benchmark and the modality selection rate metric. Our evaluation of ten representative OLLMs reveals a notable paradigm shift: unlike the ``text-dominance'' of traditional VLMs, most OLLMs exhibit a pronounced visual preference. To further understand the underlying mechanism, we conduct layer-wise probing and demonstrate that such modality preference is not static but emerges progressively in the mid-to-late layers. Building upon these insights, we leverage these internal signals to diagnose cross-modal hallucinations, achieving competitive performance across three downstream multi-modal benchmarks without task-specific data. Our work provides both a mechanistic understanding and a practical tool for building more trustworthy OLLMs. Our code and related resources are publicly available at: https://github.com/icip-cas/OmniPreference
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 18 Apr 2026.
Builds on
27 of 36 references.
References · 27
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
- 2019Language Models as Knowledge Bases?cited by 1,922
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
- 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
- 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
15 more
- 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
- 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
- 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 86
- 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
- 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
- 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
- 2024Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labelingcited by 24
- 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
- 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignmentcited by 22
- 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?cited by 22
- 2025When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMscited by 18
- 2025See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Biascited by 9
- 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflictcited by 7
- 2025XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Modelscited by 4
- 2025MaLoRA: Gated Modality LoRA for Key-Space Alignment in Multimodal LLM Fine-Tuningcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Model
- 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
- 2026AMPS: Adaptive Modality Preference Steering via Functional Entropy
- 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs
- 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination
- 2024OmniBench: Towards The Future of Universal Omni-Language Models
- 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models
- 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization