# Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts

arXiv: https://arxiv.org/abs/2505.17127 · HTML: https://mlsanity.com/paper/2505.17127/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 32)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [Future Lens: Anticipating Subsequent Tokens from a Single Hidden State](https://mlsanity.com/paper/2311.04897/) (2023)
- [The mechanistic basis of data dependence and abrupt learning in an in-context classification task](https://mlsanity.com/paper/2312.03002/) (2023)
- [VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations](https://mlsanity.com/paper/2207.00221/) (2022)
- [Probing Image-Language Transformers for Verb Understanding](https://mlsanity.com/paper/2106.09141/) (2021)
- [Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations](https://mlsanity.com/paper/2410.02762/) (2024)
- [FOIL it! Find One mismatch between Image and Language caption](https://mlsanity.com/paper/1705.01359/) (2017)
- [Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2402.18154/) (2024)
- [Reducing Hallucinations in Vision-Language Models via Latent Space Steering](https://mlsanity.com/paper/2410.15778/) (2024)
- [Competition Dynamics Shape Algorithmic Phases of In-Context Learning](https://mlsanity.com/paper/2412.01003/) (2024)
- [Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning](https://mlsanity.com/paper/2304.03916/) (2023)
- [COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs](https://mlsanity.com/paper/2309.14356/) (2023)
- [How do language models learn facts? Dynamics, curricula and hallucinations](https://mlsanity.com/paper/2503.21676/) (2025)
- [The broader spectrum of in-context learning](https://mlsanity.com/paper/2412.03782/) (2024)
- [VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark](https://mlsanity.com/paper/2403.07350/) (2024)
- [Analyzing Vision Transformers for Image Classification in Class Embedding Space](https://mlsanity.com/paper/2310.18969/) (2023)
- [PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts](https://mlsanity.com/paper/2308.01313/) (2023)
- [Forgotten Polygons: Multimodal Large Language Models are Shape-Blind](https://mlsanity.com/paper/2502.15969/) (2025)
- [Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting](https://mlsanity.com/paper/2406.00053/) (2024)

## Built on by (most cited first, 30 of 32)

- [How Do Vision-Language Models Process Conflicting Information Across Modalities?](https://mlsanity.com/paper/2507.01790/) (2025)
- [When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs](https://mlsanity.com/paper/2511.02243/) (2025)
- [Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models](https://mlsanity.com/paper/2603.19233/) (2026)
- [When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models](https://mlsanity.com/paper/2507.13868/) (2025)
- [Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts](https://mlsanity.com/paper/2604.09364/) (2026)
- [Mechanisms of Prompt-Induced Hallucination in Vision-Language Models](https://mlsanity.com/paper/2601.05201/) (2026)
- [The Persona Paradox: Medical Personas as Behavioral Priors in Clinical Language Models](https://mlsanity.com/paper/2601.05376/) (2026)
- [Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models](https://mlsanity.com/paper/2606.28273/) (2026)
- [Diagnosing Visual Ignorance in Vision-Language Models](https://mlsanity.com/paper/2606.06890/) (2026)
- [C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models](https://mlsanity.com/paper/2608.05381/) (2026)
- [Cross-Modal Taxonomic Generalization in (Vision-) Language Models](https://mlsanity.com/paper/2603.07474/) (2026)
- [When to Call an Apple Red: Humans Follow Introspective Rules, VLMs Don't](https://mlsanity.com/paper/2604.06422/) (2026)
- [CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models](https://mlsanity.com/paper/2605.17826/) (2026)
- [MATCHA: Matching Text via Contrastive Semantic Alignment](https://mlsanity.com/paper/2605.27345/) (2026)
- [Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning](https://mlsanity.com/paper/2608.06938/) (2026)
- [Which course? Discourse! Teaching Discourse and Generation in the Era of LLMs](https://mlsanity.com/paper/2602.02878/) (2026)
- [Mechanistically Interpreting Compression in Vision-Language Models](https://mlsanity.com/paper/2603.25035/) (2026)
- [Multimodal QUD: Inquisitive Questions from Scientific Figures](https://mlsanity.com/paper/2604.23733/) (2026)
- [Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery](https://mlsanity.com/paper/2605.27315/) (2026)
- [Prior Bias in Vision Language Models on UML Diagram Interpretation](https://mlsanity.com/paper/2607.02853/) (2026)
- [Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2607.26326/) (2026)
- [Prior Directions: Why GUI Grounding Gets Locked in the Past](https://mlsanity.com/paper/2607.26913/) (2026)
- [CAER: Conflict-Aware Evidence Routing with Dual Prefix Experts for Multimodal Large Language Models](https://mlsanity.com/paper/2607.28991/) (2026)
- [When Model Priors Conflict with Visual Evidence: Mitigating Commonsense-Driven Hallucinations by Selective Prior Calibration](https://mlsanity.com/paper/2607.29240/) (2026)
- [Linguistic Context Recodes Visual Representations in Vision-Language Models](https://mlsanity.com/paper/2608.00035/) (2026)
- [Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading](https://mlsanity.com/paper/2608.05675/) (2026)
- [Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition](https://mlsanity.com/paper/2609.07474/) (2026)
- [Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs](https://mlsanity.com/paper/2609.09124/) (2026)
- [(How) Do MLLMs Report Bistable Images Like Humans?](https://mlsanity.com/paper/2609.13254/) (2026)
- [Mandela-Bench: Multimodal Models Remember Canonical Images Instead of Seeing Them](https://mlsanity.com/paper/2609.32763/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) perform well on tasks such as visual question answering, but it remains unclear whether their reasoning relies more on memorized world knowledge or on the visual information present in the input image. To investigate this, we introduce Visual CounterFact, a new dataset of visually-realistic counterfactuals that put world knowledge priors (e.g, red strawberry) into direct conflict with visual input (e.g, blue strawberry). Using Visual CounterFact, we show that model predictions initially reflect memorized priors, but shift toward visual evidence in mid-to-late layers. This dynamic reveals a competition between the two modalities, with visual input ultimately overriding priors during evaluation. To control this behavior, we propose Pixels Versus Priors (PvP) steering vectors, a mechanism for controlling model outputs toward either world knowledge or visual input through activation-level interventions. On average, PvP successfully shifts 99.3% of color and 80.8% of size predictions from priors to counterfactuals. Together, these findings offer new tools for interpreting and controlling factual behavior in multimodal models.