mlsanityData to 14 Oct 2026

Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs

arXiv 2506.09047 ↗First posted 10 Jun 2025cs.CLcited by 35

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language models (VLMs) show impressive abilities to answer questions on visual inputs (e.g., counting objects in an image), yet demonstrate higher accuracies when performing an analogous task on text (e.g., counting words in a text). We investigate this accuracy gap by identifying and comparing the textitcircuits - the task-specific computational sub-graphs - in different modalities. We show that while circuits are largely disjoint between modalities, they implement relatively similar functionalities: the differences lie primarily in processing modality-specific data positions (an image or a text sequence). Zooming in on the image data representations, we observe they become aligned with the higher-performing analogous textual representations only towards later layers, too late in processing to effectively influence subsequent positions. To overcome this, we patch the representations of visual data tokens from later layers back into earlier layers. In experiments with multiple tasks and models, this simple intervention closes a third of the performance gap between the modalities, on average. Our analysis sheds light on the multi-modal performance gap in VLMs and suggests a training-free approach for reducing it.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 62 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

48 of 62 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models18 Dec
  2. 2024PaliGemma 2: A Family of Versatile VLMs for Transfer4 Dec
  3. 2024Cross-modal Information Flow in Multimodal Large Language Models27 Nov
  4. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models26 Nov
  5. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering17 Nov
  6. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
  7. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation24 Jun
  8. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Models6 Jun
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  11. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  12. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
27 more
  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  2. 2021Pay Attention to MLPscited by 368
  3. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  4. 2013Direct and Indirect Effectscited by 280
  5. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  6. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  7. 2024Pixtral 12Bcited by 179
  8. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  9. 2022Linearly Mapping from Image to Text Spacecited by 142
  10. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  11. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 121
  12. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
  13. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 113
  14. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 108
  15. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  16. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 97
  17. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  18. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  19. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  20. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  21. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 56
  22. 2023Rosetta Neurons: Mining the Common Units in a Model Zoocited by 46
  23. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 38
  24. 2024Interpreting the Second-Order Effects of Neurons in CLIPcited by 36
  25. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 28
  26. 2024Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Modelscited by 11
  27. 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 29.

35 papers.

Sort:

By area:

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  2. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?cited by 23
  3. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 15
  4. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 13
  5. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  6. 2025Map the Flow: Revealing Hidden Pathways of Information in VideoLLMscited by 10
  7. 2026Circuit Tracing in Vision-Language Models: Understanding the Internal Mechanisms of Multimodal Thinkingcited by 8
  8. 2025Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrievalcited by 7
  9. 2026Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitrationcited by 6
  10. 2026Do Language Models Track Entities Across State Changes?cited by 6
  11. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 5
  12. 2026The Dual Mechanisms of Spatial Variable Binding in Vision-Language Modelscited by 4
23 more
  1. 2026A Comprehensive Information-Decomposition Analysis of Large Vision-Language Modelscited by 4
  2. 2025Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolismcited by 3
  3. 2026Probing Visual Concepts in Lightweight Vision-Language Models for Automated Drivingcited by 3
  4. 2026Differentiable Faithfulness Alignment for Cross-Model Circuit Transfercited by 3
  5. 2026Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Modelscited by 3
  6. 2025DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Modelscited by 2
  7. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 2
  8. 2026Data-driven Circuit Discovery for Interpretability of Language Modelscited by 2
  9. 2026Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Modelscited by 1
  10. 2026From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMscited by 1
  11. 2026Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflictscited by 1
  12. 2026ProjLens: Unveiling the Role of Projectors in Multimodal Model Safetycited by 0
  13. 2026Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flowcited by 0
  14. 2026Visual Instruction Tuning Aligns Modalities through Abstractioncited by 0
  15. 2026Interleaved Speech Language Models Latently Work In Textcited by 0
  16. 2026Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sourcescited by 0
  17. 2026Attending to Multimodal Generation One Token at a Timecited by 0
  18. 2026Test-Time Training for Modality Order Consistency in Vision-Language Modelscited by 0
  19. 2026Linguistic Context Recodes Visual Representations in Vision-Language Modelscited by 0
  20. 2026S^3martCirc: Self-supervised Smart Circuit Discoverycited by 0
  21. 2026Taming CoT Obfuscation in VLMs: From Mechanistic Evidence to Activation Enforcementcited by 0
  22. 2026Virtual Encoders in Multimodal Transformerscited by 0
  23. 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Vision-Language Models Create Cross-Modal Task Representations
  2. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  3. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  4. 2025SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
  5. 2025Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring
  6. 2025Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes
  7. 2025Hidden in plain sight: VLMs overlook their visual representations
  8. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?