mlsanityData to 7 Oct 2026

Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models

arXiv 2410.03659 ↗First posted 4 Oct 2024cs.CV cs.CLcited by 28

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities for capturing and reasoning over multimodal inputs. However, these models are prone to parametric knowledge conflicts, which arise from inconsistencies of represented knowledge between their vision and language components. In this paper, we formally define the problem of cross-modality parametric knowledge conflict and present a systematic approach to detect, interpret, and mitigate them. We introduce a pipeline that identifies conflicts between visual and textual answers, showing a persistently high conflict rate across modalities in recent LVLMs regardless of the model size. We further investigate how these conflicts interfere with the inference process and propose a contrastive metric to discern the conflicting samples from the others. Building on these insights, we develop a novel dynamic contrastive decoding method that removes undesirable logits inferred from the less confident modality components based on answer confidence. For models that do not provide logits, we also introduce two prompt-based strategies to mitigate the conflicts. Our methods achieve promising improvements in accuracy on both the ViQuAE and InfoSeek datasets. Specifically, using LLaVA-34B, our proposed dynamic contrastive decoding improves an average accuracy of 2.24%.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 58 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

47 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
33 more
  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  4. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  6. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  7. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  8. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  9. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  10. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  11. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  12. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  13. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  14. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  15. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  16. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  17. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  18. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  19. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  20. 2023Context-faithful Prompting for Large Language Modelscited by 76
  21. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  22. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  23. 2024How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?cited by 44
  24. 2022Should We Rely on Entity Mentions for Relation Extraction? Debiasing Relation Extraction with Counterfactual Analysiscited by 30
  25. 2023Predicting Question-Answering Performance of Large Language Models through Semantic Consistencycited by 28
  26. 2023A Causal View of Entity Bias in (Large) Language Modelscited by 21
  27. 2022Does Your Model Classify Entities Reasonably? Diagnosing and Mitigating Spurious Correlations in Entity Typingcited by 16
  28. 2022Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extractioncited by 15
  29. 2023Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoningcited by 12
  30. 2023Self-Consistency of Large Language Models under Ambiguitycited by 12
  31. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learningcited by 10
  32. 2023How Fragile is Relation Extraction under Entity Replacements?cited by 5
  33. 2023VisAlign: Dataset for Measuring the Degree of Alignment between AI and Humans in Visual Perceptioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 17; 2026: 10.

28 papers.

Sort:

By area:

  1. 2024Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Modelscited by 57
  2. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  3. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  4. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  5. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  6. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 23
  7. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?cited by 22
  8. 2025Benchmarking Multimodal Knowledge Conflict for Large Multimodal Modelscited by 18
  9. 2025Diagnosing and Mitigating Modality Interference in Multimodal Large Language Modelscited by 18
  10. 2025When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMscited by 18
  11. 2025Robust Multimodal Large Language Models Against Modality Conflictcited by 17
  12. 2025Mitigating Modal Imbalance in Multimodal Reasoningcited by 8
16 more
  1. 2025Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaborationcited by 5
  2. 2025OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoningcited by 5
  3. 2026Learning Adaptive Reasoning Paths for Efficient Visual Reasoningcited by 5
  4. 2025Is Extending Modality The Right Path Towards Omni-Modality?cited by 4
  5. 2025CLASH: A Benchmark for Cross-Modal Contradiction Detectioncited by 2
  6. 2025Data Metabolism: An Efficient Data Design Schema For Vision Language Modelcited by 1
  7. 2025What Color Is the Text? A Benchmark for Hallucination Induced by Image-Embedded Promptcited by 1
  8. 2026ProGAL-VLA: Grounded Alignment through Prospective Reasoning in Vision-Language-Action Modelscited by 1
  9. 2026Evidence-Grounded Forensic Reasoning for Detecting and Grounding Multi-Modal Media Manipulationcited by 1
  10. 2026R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoningcited by 0
  11. 2026BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QAcited by 0
  12. 2026RIHA: Report-Image Hierarchical Alignment for Radiology Report Generationcited by 0
  13. 2026ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?cited by 0
  14. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Biascited by 0
  15. 2026ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoningcited by 0
  16. 2026Same evidence, different judgments: Evidence noncommutative in vision/speech-text conflictscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs
  2. 2024Unveiling the Tapestry of Consistency in Large Vision-Language Models
  3. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability
  4. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  5. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models
  6. 2026Which Source Wins? Task-Dependent Reliance in Vision-Language Models
  7. 2024CAST: Cross-modal Alignment Similarity Test for Vision Language Models
  8. 2025CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution