mlsanityData to 14 Oct 2026

Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts

arXiv 2505.17127 ↗First posted 21 May 2025cs.CV cs.LGcited by 32

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) perform well on tasks such as visual question answering, but it remains unclear whether their reasoning relies more on memorized world knowledge or on the visual information present in the input image. To investigate this, we introduce Visual CounterFact, a new dataset of visually-realistic counterfactuals that put world knowledge priors (e.g, red strawberry) into direct conflict with visual input (e.g, blue strawberry). Using Visual CounterFact, we show that model predictions initially reflect memorized priors, but shift toward visual evidence in mid-to-late layers. This dynamic reveals a competition between the two modalities, with visual input ultimately overriding priors during evaluation. To control this behavior, we propose Pixels Versus Priors (PvP) steering vectors, a mechanism for controlling model outputs toward either world knowledge or visual input through activation-level interventions. On average, PvP successfully shifts 99.3% of color and 80.8% of size predictions from priors to counterfactuals. Together, these findings offer new tools for interpreting and controlling factual behavior in multimodal models.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 51 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

32 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations1 Jul

Other references · 31

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  4. 2023Steering Language Models With Activation Engineeringcited by 922
  5. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  6. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  7. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  10. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  11. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  12. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
19 more
  1. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
  2. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  3. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  4. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  5. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  6. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 91
  7. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 59
  8. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 54
  9. 2023Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningcited by 47
  10. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
  11. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 37
  12. 2024The broader spectrum of in-context learningcited by 35
  13. 2024VLKEB: A Large Vision-Language Model Knowledge Editing Benchmarkcited by 30
  14. 2023Analyzing Vision Transformers for Image Classification in Class Embedding Spacecited by 26
  15. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contextscited by 17
  16. 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blindcited by 16
  17. 2024Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgettingcited by 15
  18. 2022Spuriosity Rankings: Sorting Data to Measure and Mitigate Biasescited by 13
  19. 2024Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledgecited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 29.

32 papers.

Sort:

By area:

  1. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?cited by 23
  2. 2025When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMscited by 21
  3. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Modelscited by 15
  4. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  5. 2026Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflictscited by 11
  6. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 5
  7. 2026The Persona Paradox: Medical Personas as Behavioral Priors in Clinical Language Modelscited by 3
  8. 2026Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Modelscited by 3
  9. 2026Diagnosing Visual Ignorance in Vision-Language Modelscited by 2
  10. 2026C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Modelscited by 2
  11. 2026Cross-Modal Taxonomic Generalization in (Vision-) Language Modelscited by 1
  12. 2026When to Call an Apple Red: Humans Follow Introspective Rules, VLMs Don'tcited by 1
20 more
  1. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  2. 2026MATCHA: Matching Text via Contrastive Semantic Alignmentcited by 1
  3. 2026Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoningcited by 1
  4. 2026Which course? Discourse! Teaching Discourse and Generation in the Era of LLMscited by 0
  5. 2026Mechanistically Interpreting Compression in Vision-Language Modelscited by 0
  6. 2026Multimodal QUD: Inquisitive Questions from Scientific Figurescited by 0
  7. 2026Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagerycited by 0
  8. 2026Prior Bias in Vision Language Models on UML Diagram Interpretationcited by 0
  9. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Modelscited by 0
  10. 2026Prior Directions: Why GUI Grounding Gets Locked in the Pastcited by 0
  11. 2026CAER: Conflict-Aware Evidence Routing with Dual Prefix Experts for Multimodal Large Language Modelscited by 0
  12. 2026When Model Priors Conflict with Visual Evidence: Mitigating Commonsense-Driven Hallucinations by Selective Prior Calibrationcited by 0
  13. 2026Linguistic Context Recodes Visual Representations in Vision-Language Modelscited by 0
  14. 2026Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Readingcited by 0
  15. 2026Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognitioncited by 0
  16. 2026Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMscited by 0
  17. 2026(How) Do MLLMs Report Bistable Images Like Humans?cited by 0
  18. 2026Mandela-Bench: Multimodal Models Remember Canonical Images Instead of Seeing Themcited by 0
  19. 2026In-Context Learning Can Help Vision-Language Models Overcome Training Priorscited by 0
  20. 2026Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
  2. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  3. 2025Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understanding
  4. 2025CF-VLM:CounterFactual Vision-Language Fine-tuning
  5. 2026Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models
  6. 2023What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models
  7. 2026MM-OPD: Towards One More Bottleneck Between Perception and Reasoning
  8. 2026Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning