mlsanityData to 7 Oct 2026

Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models

arXiv 2604.16902 ↗First posted 18 Apr 2026cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Native Omni-modal Large Language Models (OLLMs) have shifted from pipeline architectures to unified representation spaces. However, this native integration gives rise to a critical yet underexplored phenomenon: modality preference. To bridge this gap, we first systematically quantify modality preference of OLLMs using a newly-curated conflict-based benchmark and the modality selection rate metric. Our evaluation of ten representative OLLMs reveals a notable paradigm shift: unlike the ``text-dominance'' of traditional VLMs, most OLLMs exhibit a pronounced visual preference. To further understand the underlying mechanism, we conduct layer-wise probing and demonstrate that such modality preference is not static but emerges progressively in the mid-to-late layers. Building upon these insights, we leverage these internal signals to diagnose cross-modal hallucinations, achieving competitive performance across three downstream multi-modal benchmarks without task-specific data. Our work provides both a mechanistic understanding and a practical tool for building more trustworthy OLLMs. Our code and related resources are publicly available at: https://github.com/icip-cas/OmniPreference

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 18 Apr 2026.

Builds on

27 of 36 references.

References · 27

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  10. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  11. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  12. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
15 more
  1. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  2. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  3. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 86
  4. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  5. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  6. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  7. 2024Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labelingcited by 24
  8. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  9. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignmentcited by 22
  10. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?cited by 22
  11. 2025When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMscited by 18
  12. 2025See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Biascited by 9
  13. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflictcited by 7
  14. 2025XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Modelscited by 4
  15. 2025MaLoRA: Gated Modality LoRA for Key-Space Alignment in Multimodal LLM Fine-Tuningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Model
  2. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
  3. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropy
  4. 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs
  5. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination
  6. 2024OmniBench: Towards The Future of Universal Omni-Language Models
  7. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models
  8. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization