# What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation

arXiv: https://arxiv.org/abs/2406.16320 · HTML: https://mlsanity.com/paper/2406.16320/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 38 of its 61 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)
- Mechanistic interpretability / [Interpreting vision and multimodal models](https://mlsanity.com/mechinterp/interpreting-vision-and-multimodal-models/)

## Builds on (most cited first, 30 of 38)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning](https://mlsanity.com/paper/2110.13214/) (2021)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [Probing Image-Language Transformers for Verb Understanding](https://mlsanity.com/paper/2106.09141/) (2021)
- [Circuit Component Reuse Across Tasks in Transformer Language Models](https://mlsanity.com/paper/2310.08744/) (2023)
- [Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information](https://mlsanity.com/paper/1808.08079/) (2018)
- [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020)
- [FOIL it! Find One mismatch between Image and Language caption](https://mlsanity.com/paper/1705.01359/) (2017)
- [Successor Heads: Recurring, Interpretable Attention Heads In The Wild](https://mlsanity.com/paper/2312.09230/) (2023)
- [Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers](https://mlsanity.com/paper/2109.04448/) (2021)
- [Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!](https://mlsanity.com/paper/2010.06572/) (2020)

## Built on by (most cited first, 30 of 30)

- [Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs](https://mlsanity.com/paper/2506.09047/) (2025)
- [Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning](https://mlsanity.com/paper/2504.02922/) (2025)
- [How Do Vision-Language Models Process Conflicting Information Across Modalities?](https://mlsanity.com/paper/2507.01790/) (2025)
- [Forgotten Polygons: Multimodal Large Language Models are Shape-Blind](https://mlsanity.com/paper/2502.15969/) (2025)
- [Visual symbolic mechanisms: Emergent symbol processing in vision language models](https://mlsanity.com/paper/2506.15871/) (2025)
- [How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding](https://mlsanity.com/paper/2508.20279/) (2025)
- [V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models](https://mlsanity.com/paper/2509.14837/) (2025)
- [Interpretable Debiasing of Vision-Language Models for Social Fairness](https://mlsanity.com/paper/2602.24014/) (2026)
- [Causal Tracing of Audio-Text Fusion in Large Audio Language Models](https://mlsanity.com/paper/2603.13768/) (2026)
- [Gaze Heads: How VLMs Look at What They Describe](https://mlsanity.com/paper/2606.14703/) (2026)
- [Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering](https://mlsanity.com/paper/2603.06854/) (2026)
- [Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Models](https://mlsanity.com/paper/2510.21740/) (2025)
- [Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving](https://mlsanity.com/paper/2603.06054/) (2026)
- [Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-Language Models](https://mlsanity.com/paper/2604.07802/) (2026)
- [Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication](https://mlsanity.com/paper/2506.22274/) (2025)
- [MIMIC: Multimodal Inversion for Model Interpretation and Conceptualization](https://mlsanity.com/paper/2508.07833/) (2025)
- [When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities](https://mlsanity.com/paper/2607.08605/) (2026)
- [What Do They See? Interpreting Complex Road Scenarios Through the Eyes of Vision-Language-Action Models for Safe and Trustworthy Autonomous Vehicle Learning](https://mlsanity.com/paper/2607.16938/) (2026)
- [CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models](https://mlsanity.com/paper/2609.38810/) (2026)
- [Beyond the Linear Separability Ceiling: Aligning Representations in VLMs](https://mlsanity.com/paper/2507.07574/) (2025)
- [Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models](https://mlsanity.com/paper/2602.22469/) (2026)
- [Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment](https://mlsanity.com/paper/2604.00913/) (2026)
- [Multimodal QUD: Inquisitive Questions from Scientific Figures](https://mlsanity.com/paper/2604.23733/) (2026)
- [Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow](https://mlsanity.com/paper/2605.21980/) (2026)
- [How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA](https://mlsanity.com/paper/2607.16094/) (2026)
- [ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models](https://mlsanity.com/paper/2607.20092/) (2026)
- [Prior Directions: Why GUI Grounding Gets Locked in the Past](https://mlsanity.com/paper/2607.26913/) (2026)
- [From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists](https://mlsanity.com/paper/2609.21543/) (2026)
- [Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoning](https://mlsanity.com/paper/2609.35486/) (2026)
- [What Do Visual-Head Discovery Scores Actually Find? Attention Source, Necessity, and Random-Control Pitfalls in Vision-Language Models](https://mlsanity.com/paper/2610.13111/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) have gained community-spanning prominence due to their ability to integrate visual and textual inputs to perform complex tasks. Despite their success, the internal decision-making processes of these models remain opaque, posing challenges in high-stakes applications. To address this, we introduce NOTICE, the first Noise-free Text-Image Corruption and Evaluation pipeline for mechanistic interpretability in VLMs. NOTICE incorporates a Semantic Minimal Pairs (SMP) framework for image corruption and Symmetric Token Replacement (STR) for text. This approach enables semantically meaningful causal mediation analysis for both modalities, providing a robust method for analyzing multimodal integration within models like BLIP. Our experiments on the SVO-Probes, MIT-States, and Facial Expression Recognition datasets reveal crucial insights into VLM decision-making, identifying the significant role of middle-layer cross-attention heads. Further, we uncover a set of ``universal cross-attention heads'' that consistently contribute across tasks and modalities, each performing distinct functions such as implicit image segmentation, object inhibition, and outlier inhibition. This work paves the way for more transparent and interpretable multimodal systems.