mlsanityData to 14 Oct 2026

Source-Modality Monitoring in Vision-Language Models

arXiv 2604.22038 ↗First posted 23 Apr 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We define and investigate source-modality monitoring -- the ability of multimodal models to track and communicate the input source from which pieces of information originate. We consider source-modality monitoring as an instance of the more general binding problem, and evaluate the extent to which models exploit syntactic vs. semantic signals in order to bind words like image in a user-provided prompt to specific components of their input and context (i.e., actual images). Across experiments spanning 11 vision-language models (VLMs) performing target-modality information retrieval tasks, we find that both syntactic and semantic signals play an important role, but that the latter tend to outweigh the former in cases when modalities are highly distinct distributionally. We discuss the implications of these findings for model robustness, and in the context of increasingly multimodal agentic systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 43 references that are in the map. First posted on arXiv 23 Apr 2026.

Builds on

36 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?2 Jul
  2. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict11 Apr
  3. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar

Other references · 33

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  9. 2025Gemma 3 Technical Reportcited by 1,824
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
21 more
  1. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  3. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  4. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  5. 2019Designing and Interpreting Probes with Control Taskscited by 640
  6. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  7. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  8. 2023In-Context Learning Creates Task Vectorscited by 312
  9. 2023Function Vectors in Large Language Modelscited by 308
  10. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
  11. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  12. 2022Language models show human-like content effects on reasoning taskscited by 173
  13. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  14. 2023How do Language Models Bind Entities in Context?cited by 108
  15. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 84
  16. 2025Embodied AI Agents: Modeling the Worldcited by 61
  17. 2024Source-Aware Training Enables Knowledge Attribution in Language Modelscited by 30
  18. 2025Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Contextcited by 26
  19. 2025How Do Transformers Learn Variable Binding in Symbolic Programs?cited by 21
  20. 2025When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMscited by 21
  21. 2025Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spotscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models
  2. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs
  3. 2026Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
  4. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem
  5. 2026Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts
  6. 2026V-Mem: Modality-Routed Retrieval for Long-Term Multimodal Agentic Memory
  7. 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
  8. 2026ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models