mlsanityData to 14 Oct 2026

Visual Access Boundaries in Vision-Language Model Reasoning

arXiv 2607.12815 ↗First posted 14 Jul 2026cs.AIcited by 2

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

When do vision-language models need direct access to the image while generating an answer? We study image dependence during answer generation by examining how the visual information needed for the current question becomes available in context. We intervene on direct image access while retaining previously computed states. Across real-image and synthetic tasks, we show that, depending on the generation process, direct access can continue to support accuracy after question processing. Supplying the required attributes as text in the context weakens this dependence. On synthetic tasks, we also examine how dependence changes as the model itself states the required attributes. Before attribute expression, severing access reduces accuracy, and replacing image-side states shifts answers toward the counterfactual content. After sufficient expression, both interventions have smaller effects. Even with an identical generated prefix, dependence differs according to whether image access was available during question processing. Thus, both the visible text and the preceding image access matter. Several of these patterns hold across model families, including Qwen2.5-VL-32B and InternVL3-14B. These findings offer a view of image dependence in terms of the information needed for the current question and the history of image access, beyond generation position alone. This perspective provides a basis for deciding when to reduce visual access during an answer and which visual information to retain for subsequent questions.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 54 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

38 of 54 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026On Test-Time Scaling for Vision-Language Models27 Jun
  2. 2026How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning17 Feb
  3. 2025Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Information29 May
  4. 2025Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning17 Mar
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  6. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr

Other references · 32

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  7. 2019Attention is not Explanationcited by 929
  8. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  9. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  10. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  11. 2019Attention is not not Explanationcited by 636
  12. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
20 more
  1. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  2. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  3. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  4. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  5. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  6. 2024How to use and interpret activation patchingcited by 214
  7. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 191
  8. 2018TallyQA: Answering Complex Counting Questionscited by 153
  9. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 78
  10. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  11. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 68
  12. 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 67
  13. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 43
  14. 2025Visual Representations inside the Language Modelcited by 15
  15. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  16. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 14
  17. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 13
  18. 2026Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examinationcited by 13
  19. 2025Hidden in plain sight: VLMs overlook their visual representationscited by 12
  20. 2026Pathways of Visual Information Flow in Vision-Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structurescited by 0
  2. 2026Certified Interface Aliases: Exact Collisions in Vision-Language Preprocessing, and When They Existcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalization
  2. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision
  3. 2025DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Models
  4. 2025MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models
  5. 2026Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
  6. 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
  7. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models
  8. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning