mlsanityData to 7 Oct 2026

Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs

arXiv 2601.19202 ↗First posted 27 Jan 2026cs.CLcited by 6

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have shown strong multimodal reasoning capabilities on Visual-Question-Answering (VQA) benchmarks. However, their robustness against textual misinformation remains under-explored. While existing research has studied the effect of misinformation in text-only domains, it is not clear how VLMs arbitrate between contradictory information from different modalities. To bridge the gap, we first propose the CONTEXT-VQA (i.e., Conflicting Text) dataset, consisting of image-question pairs together with systematically generated persuasive prompts that deliberately conflict with visual evidence. Then, a thorough evaluation framework is designed and executed to benchmark the susceptibility of various models to these conflicting multimodal inputs. Comprehensive experiments over 11 state-of-the-art VLMs reveal that these models are indeed vulnerable to misleading textual prompts, often overriding clear visual evidence in favor of the conflicting text, and show an average performance drop of over 48.2% after only one round of persuasive conversation. Our findings highlight a critical limitation in current VLMs and underscore the need for improved robustness against textual manipulation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 63 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

39 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs12 Jun

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  8. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  10. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  11. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  12. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
26 more
  1. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  2. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  3. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  4. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  5. 2023Are aligned neural networks adversarially aligned?cited by 305
  6. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  7. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  8. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  9. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 107
  10. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  11. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 50
  12. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignmentcited by 42
  13. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
  14. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 38
  15. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 37
  16. 2023Exploring and Exploiting Decision Boundary Dynamics for Adversarial Robustnesscited by 35
  17. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Modelscited by 31
  18. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Studycited by 28
  19. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspectivecited by 28
  20. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  21. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  22. 2025Cautious Weight Decaycited by 12
  23. 2024Memory-Efficient Optimization with Factorized Hamiltonian Descentcited by 11
  24. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approachcited by 7
  25. 2024VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognitioncited by 7
  26. 2025E2LVLM:Evidence-Enhanced Large Vision-Language Model for Multimodal Out-of-Context Misinformation Detectioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Visual Credit Audit for Multimodal Spatial Reasoningcited by 1
  2. 2026SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Modelscited by 1
  3. 2026Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictionscited by 0
  4. 2026Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagerycited by 0
  5. 2026Text Over Image: Auditing Multimodal Robustness in Synthetic Medical Image Detectioncited by 0
  6. 2026Same evidence, different judgments: Evidence noncommutative in vision/speech-text conflictscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
  2. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  3. 2026When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models
  4. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios
  5. 2025Questioning the Stability of Visual Question Answering
  6. 2025Evaluating Variance in Visual Question Answering Benchmarks
  7. 2025VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering
  8. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark