mlsanityData to 14 Oct 2026

Challenges in Understanding Modality Conflict in Vision-Language Models

arXiv 2509.02805 ↗First posted 2 Sep 2025cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

This paper highlights the challenge of decomposing conflict detection from conflict resolution in Vision-Language Models (VLMs) and presents potential approaches, including using a supervised metric via linear probes and group-based attention pattern analysis. We conduct a mechanistic investigation of LLaVA-OV-7B, a state-of-the-art VLM that exhibits diverse resolution behaviors when faced with conflicting multimodal inputs. Our results show that a linearly decodable conflict signal emerges in the model's intermediate layers and that attention patterns associated with conflict detection and resolution diverge at different stages of the network. These findings support the hypothesis that detection and resolution are functionally distinct mechanisms. We discuss how such decomposition enables more actionable interpretability and targeted interventions for improving model robustness in challenging multimodal settings.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 10 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

8 of 10 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar

Other references · 7

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2022In-context Learning and Induction Headscited by 964
  3. 2025Open Problems in Mechanistic Interpretabilitycited by 179
  4. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  5. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priorscited by 37
  6. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  7. 2024From Insights to Actions: The Impact of Interpretability and Analysis Research on NLPcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflictcited by 2
  2. 2026Causal Evidence for Attention Head Imbalance in Modality Conflict Hallucinationcited by 1
  3. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Biascited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models
  2. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  3. 2025SegSub: Evaluating Robustness to Knowledge Conflicts and Hallucinations in Vision-Language Models
  4. 2026Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
  5. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs
  6. 2025Robust Multimodal Large Language Models Against Modality Conflict
  7. 2025Conflict Adaptation in Vision-Language Models
  8. 2026Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts