mlsanityData to 7 Oct 2026

Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts

arXiv 2609.00293 ↗First posted 31 Aug 2026cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We investigate how vision-language models (VLMs) handle context-memory conflicts; that is, situations in which the model is given information in context that differs from what was stored parametrically during training. We document asymmetric biases: models tend to prefer in-context information about entities which appear in text, but prefer parametric information about entities which appear in images. We relate this asymmetry to the late representational alignment across modalities, showing that the longer processing time associated with resolving visual entities prevents the suppression of the model's usual factual recall mechanism, thus resulting in more parametric answers. Chain-of-thought reasoning does not appear to resolve the gap, but increasing the amount of visual information in the context does show an effect. These results illustrate the complexity of ensuring consistent behavior as models become increasingly multimodal and retrieval-augmented.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 31 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

17 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs10 Jun

Other references · 16

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  4. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  5. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  6. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 87
  7. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  8. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  9. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 60
  10. 2024Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMscited by 39
  11. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMscited by 37
  12. 2024Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documentscited by 27
4 more
  1. 2025Taming Knowledge Conflicts in Language Modelscited by 27
  2. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMscited by 24
  3. 2025MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledgecited by 22
  4. 2025Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrievalcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Using OCR Heads to Verbalize Image Semanticscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  2. 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
  3. 2026Which Source Wins? Task-Dependent Reliance in Vision-Language Models
  4. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models
  5. 2026ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models
  6. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?
  7. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
  8. 2026Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models