mlsanityData to 14 Oct 2026

Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs

arXiv 2510.17771 ↗First posted 20 Oct 2025cs.AI cs.CVcited by 55

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) achieve strong results on multimodal tasks such as visual question answering, yet they can still fail even when the correct visual evidence is present. In this work, we systematically investigate whether these failures arise from not perceiving the evidence or from not leveraging it effectively. By examining layer-wise attention dynamics, we find that shallow layers focus primarily on text, while deeper layers sparsely but reliably attend to localized evidence regions. Surprisingly, VLMs often perceive the visual evidence when outputting incorrect answers, a phenomenon we term ``seeing but not believing'' that widely exists in major VLM families. Building on this, we introduce an inference-time intervention that highlights deep-layer evidence regions through selective attention-based masking. It requires no training and consistently improves accuracy across multiple families, including LLaVA, Qwen, Gemma, and InternVL. These results show that VLMs encode reliable evidence internally but under-utilize it, making such signals explicit can bridge the gap between perception and reasoning, advancing the diagnostic understanding and reliability of VLMs.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 37 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

34 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar
  2. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Models13 Jun
  3. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2025Gemma 3 Technical Reportcited by 1,824
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  12. 2021InfographicVQAcited by 437
19 more
  1. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  2. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  3. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  4. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  5. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 113
  6. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 95
  7. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  8. 2024Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Modelscited by 58
  9. 2024Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?cited by 48
  10. 2024MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generationcited by 44
  11. 2024VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Informationcited by 39
  12. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Trainingcited by 38
  13. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 30
  14. 2024Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documentscited by 27
  15. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 26
  16. 2024Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generationcited by 12
  17. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generationcited by 11
  18. 2025On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generationcited by 10
  19. 2025Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 53.

55 papers.

Sort:

By area:

  1. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 50
  2. 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
  3. 2026Visually-Guided Policy Optimization for Multimodal Reasoningcited by 11
  4. 2026Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusionscited by 8
  5. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 7
  6. 2026Toward Cognitive Supersensing in Multimodal Large Language Modelcited by 7
  7. 2026Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Modelscited by 6
  8. 2026Do MLLMs Really See It: Reinforcing Visual Attention in Multimodal LLMscited by 4
  9. 2026Mixture of Sequence: Theme-Aware Mixture-of-Experts for Long-Sequence Recommendationcited by 4
  10. 2026Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steeringcited by 3
  11. 2026VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understandingcited by 3
  12. 2026Probing Visual Concepts in Lightweight Vision-Language Models for Automated Drivingcited by 3
38 more
  1. 2025Continual Low-Rank Adapters for LLM-based Generative Recommender Systemscited by 2
  2. 2026SPD-Faith Bench: Diagnosing and Improving Faithfulness in Chain-of-Thought for Multimodal Large Language Modelscited by 2
  3. 2026Don't Blink: Evidence Collapse during Multimodal Reasoningcited by 2
  4. 2026OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Taskscited by 2
  5. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 2
  6. 2026Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoningcited by 2
  7. 2026The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidencecited by 2
  8. 2026Graph homophily booster: Reimagining the role of discrete features in heterophilic graph learningcited by 1
  9. 2026When to Call an Apple Red: Humans Follow Introspective Rules, VLMs Don'tcited by 1
  10. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  11. 2026MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cuescited by 1
  12. 2026Detect Before You Leap: Mirage Detection in Vision-Language Modelscited by 1
  13. 2026Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selectioncited by 1
  14. 2026Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehensioncited by 1
  15. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigationcited by 1
  16. 2026Knowing Isn't Always Saying: When Do Spatial Encodings Reach Answers in Vision-Language Models?cited by 1
  17. 2026System-Mediated Attention Imbalances Make Vision-Language Models Say Yescited by 0
  18. 2026Auditing Disability Representation in Vision-Language Modelscited by 0
  19. 2026NAST: Improving Negation Handling in Medical Vision-Language Models through Negation-Aware Selective Trainingcited by 0
  20. 2026DISC: Decoupling Instruction from State-Conditioned Control via Policy Generationcited by 0
  21. 2026Encoded but Not Routed: Explaining the Table-Chart Gap in Scientific Claim Verificationcited by 0
  22. 2026Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoderscited by 0
  23. 2026It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scalingcited by 0
  24. 2026SAVER: Selective Auditing of Verbal Evidence for Error Recovery in VLM Change Reasoningcited by 0
  25. 2026EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supportscited by 0
  26. 2026Mechanism-Level Evaluation for Vision-Language Models: Controlled Activation-Replacement Diagnosis of Gender Biascited by 0
  27. 2026Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMscited by 0
  28. 2026Certified Interface Aliases: Exact Collisions in Vision-Language Preprocessing, and When They Existcited by 0
  29. 2026Seeing and Solving Are Not Enough for Vision-Language Modelscited by 0
  30. 2026Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidencecited by 0
  31. 2026Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Modelscited by 0
  32. 2026On-Policy Visual Evidence Distillationcited by 0
  33. 2026Beyond Attention Imbalance: Mitigating Hallucinations via Spectral Surgerycited by 0
  34. 2026When Can Text Replace Vision? Structural Bottlenecks in Diagram Reasoningcited by 0
  35. 2026Towards Reliable Vision-Language Models for Autonomous Drivingcited by 0
  36. 2026Rotated, but How Far? Diagnosing and Improving Object-Rotation Reasoning in VLMscited by 0
  37. 2026Visual Evidence Under Cross-Examination: Evaluating and Controlling Decision-Level Evidence Use in Vision-Language Modelscited by 0
  38. 2026OccludeBench: Can VLMs Find What's Missing?cited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Evidence-RL: Towards Evidence-intensive Visual Reasoning
  2. 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
  3. 2026Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning
  4. 2026Diagnosing Visual Ignorance in Vision-Language Models
  5. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs
  6. 2026Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
  7. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow
  8. 2025Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning