mlsanityData to 14 Oct 2026

When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models

arXiv 2507.13868 ↗First posted 18 Jul 2025cs.CV cs.AIcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) increasingly combine visual and textual information to perform complex tasks. However, conflicts between their internal knowledge and external visual input can lead to hallucinations and unreliable predictions. In this work, we investigate the mechanisms that VLMs use to resolve cross-modal conflicts by introducing WHOOPS-AHA!, a dataset of multimodal counterfactual queries that deliberately contradict internal commonsense knowledge. Through logit inspection, we identify a small set of attention heads that mediate this conflict. By intervening in these heads, we can steer the model towards its internal parametric knowledge or the visual information. Our results show that attention patterns on these heads effectively locate image regions that influence visual overrides, providing a more precise attribution compared to gradient-based methods.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 58 references that are in the map. First posted on arXiv 18 Jul 2025.

Builds on

36 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?2 Jul
  2. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs10 Jun
  3. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts21 May

Other references · 33

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2025Gemma 3 Technical Reportcited by 1,824
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  5. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  8. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  9. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  10. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  12. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
21 more
  1. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  2. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  3. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  4. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 144
  5. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  6. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  7. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  8. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 91
  9. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 81
  10. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  11. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  12. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  13. 2023Characterizing Mechanisms for Factual Recall in Language Modelscited by 48
  14. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  15. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 45
  16. 2024Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactualscited by 40
  17. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
  18. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMscited by 38
  19. 2024Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMscited by 22
  20. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
  21. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergencecited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2025Head Pursuit: Probing Attention Specialization in Multimodal Transformerscited by 17
  2. 2025Conflict Adaptation in Vision-Language Modelscited by 1
  3. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  4. 2026Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoningcited by 1
  5. 2026Transcoders Trace Visual Grounding and Hallucinations in Vision-Language Modelscited by 0
  6. 2026Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sourcescited by 0
  7. 2026Prior Directions: Why GUI Grounding Gets Locked in the Pastcited by 0
  8. 2026GeoArbiter: Verifiability-Guided Grounding for Remote-Sensing Multimodal LLMscited by 0
  9. 2026Which Source Wins? Task-Dependent Reliance in Vision-Language Modelscited by 0
  10. 2026MoGround: Measuring and Mitigating Modality Distraction in Vision-Language Modelscited by 0
  11. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models
  2. 2026Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models
  3. 2025SegSub: Evaluating Robustness to Knowledge Conflicts and Hallucinations in Vision-Language Models
  4. 2026Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts
  5. 2025GLIMPSE: Holistic Cross-Modal Explainability for Large Vision-Language Models
  6. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  7. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  8. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs