mlsanityData to 7 Oct 2026

Words or Vision: Do Vision-Language Models Have Blind Faith in Text?

arXiv 2503.02199 ↗First posted 4 Mar 2025cs.CV cs.AI cs.CLcited by 86

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) excel in integrating visual and textual information for vision-centric tasks, but their handling of inconsistencies between modalities is underexplored. We investigate VLMs' modality preferences when faced with visual data and varied textual inputs in vision-centered settings. By introducing textual variations to four vision-centric tasks and evaluating ten Vision-Language Models (VLMs), we discover a ``blind faith in text'' phenomenon: VLMs disproportionately trust textual data over visual data when inconsistencies arise, leading to significant performance drops under corrupted text and raising safety concerns. We analyze factors influencing this text bias, including instruction prompts, language model size, text relevance, token order, and the interplay between visual and textual certainty. While certain factors, such as scaling up the language model size, slightly mitigate text bias, others like token order can exacerbate it due to positional biases inherited from language models. To address this issue, we explore supervised fine-tuning with text augmentation and demonstrate its effectiveness in reducing text bias. Additionally, we provide a theoretical analysis suggesting that the blind faith in text phenomenon may stem from an imbalance of pure text and multi-modal data during training. Our findings highlight the need for balanced training and careful consideration of modality interactions in VLMs to enhance their robustness and reliability in handling multi-modal data inconsistencies.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 46 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

35 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  3. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
20 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  4. 2023A Survey on Multimodal Large Language Modelscited by 898
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  6. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  7. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  8. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  9. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  10. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  11. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  12. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  13. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  14. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  15. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  16. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  17. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  18. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 109
  19. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  20. 2024KnowPhish: Large Language Models Meet Multimodal Knowledge Graphs for Enhancing Reference-Based Phishing Detectioncited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 57.

86 papers.

Sort:

By area:

  1. 2025Latent Visual Reasoningcited by 99
  2. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  3. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?cited by 22
  4. 2026Large Language Model Reasoning Failurescited by 20
  5. 2025When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMscited by 18
  6. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  7. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 14
  8. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 13
  9. 2025When Audio and Text Disagree: Revealing Text Bias in Large Audio-Language Modelscited by 11
  10. 2026Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusionscited by 8
  11. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflictcited by 7
  12. 2026More Images, More Problems? A Controlled Analysis of VLM Failure Modescited by 7
38 more
  1. 2025Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraintscited by 6
  2. 2026V-FAT: Benchmarking Visual Fidelity Against Text-biascited by 6
  3. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
  4. 2025When Silence Matters: The Impact of Irrelevant Audio on Text Reasoning in Large Audio-Language Modelscited by 4
  5. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 4
  6. 2026Tone Matters: The Impact of Linguistic Tone on Hallucination in VLMscited by 4
  7. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  8. 2026Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillationcited by 4
  9. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Abilitycited by 3
  10. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  11. 2025Challenges in Understanding Modality Conflict in Vision-Language Modelscited by 3
  12. 2025Towards Fine-Grained Human Motion Video Captioningcited by 3
  13. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Modelscited by 3
  14. 2025MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answeringcited by 3
  15. 2026Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gaincited by 3
  16. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 3
  17. 2026LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Modelscited by 3
  18. 2026WARD: Adversarially Robust Defense of Web Agents Against Prompt Injectionscited by 3
  19. 2025CLASH: A Benchmark for Cross-Modal Contradiction Detectioncited by 2
  20. 2025Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answeringcited by 2
  21. 2026FINER: MLLMs Hallucinate under Fine-grained Negative Queriescited by 2
  22. 2026Graph-to-Frame RAG: Visual-Space Knowledge Fusion for Training-Free and Auditable Video Reasoningcited by 2
  23. 2026Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Modelscited by 2
  24. 2026Diagnosing Visual Ignorance in Vision-Language Modelscited by 2
  25. 2025Text-conditioned State Space Model For Domain-generalized Change Detection Visual Question Answeringcited by 1
  26. 2025What Color Is the Text? A Benchmark for Hallucination Induced by Image-Embedded Promptcited by 1
  27. 2026Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoningcited by 1
  28. 2026Do Vision Language Models Need to Process Image Tokens?cited by 1
  29. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidancecited by 1
  30. 2026Medical Context Distorts Decisions in Clinical Vision Language Modelscited by 1
  31. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  32. 2026Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphscited by 1
  33. 2026From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQAcited by 1
  34. 2025Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structurecited by 0
  35. 2025Using Perspectival Words Is Harder Than Vocabulary Words for Humans and Even More So for Multimodal Language Modelscited by 0
  36. 2025Innocence in the Crossfire: Roles of Skip Connections in Jailbreaking Visual Language Modelscited by 0
  37. 2025A Video Is Not Worth a Thousand Wordscited by 0
  38. 2025Temporal-Visual Semantic Alignment: A Unified Architecture for Transferring Spatial Priors from Vision Models to Zero-Shot Temporal Taskscited by 0

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  2. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  3. 2026VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?
  4. 2025Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models
  5. 2026Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
  6. 2025Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes
  7. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
  8. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs