mlsanityData to 7 Oct 2026

Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination

arXiv 2605.15864 ↗First posted 15 May 2026cs.CV cs.CLcited by 12

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) often produce self-reflective statements like "let me check the figure again" during reasoning. Do such statements trigger genuine visual re-examination, or are they merely learned textual patterns? We investigate this via VisualSwap, an image-swap probing framework: after a model reasons over an image, we replace it with a visually similar but semantically different one and test whether the model notices. We introduce VS-Bench, 800 image pairs curated from MathVista, MathVerse, MathVision, and MMMU-Pro. Experiments on Qwen3-VL, Kimi-VL, and ERNIE-VL reveal a striking failure: models overwhelmingly miss the swap, with accuracy dropping by up to 60%. Counterintuitively, thinking models are nearly 3x more vulnerable than their instructed counterparts, and scaling offers no mitigation. Multi-turn user instructions restore visual grounding, but self-generated reflective statements during continuous generation do not. Attention analysis explains why: user instructions substantially elevate attention to visual tokens, whereas self-reflection does not. Current VLMs tend to say rather than actually see when claiming to perform visual re-examination. Our code and dataset are available at the project page: https://visualswap.github.io

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 45 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

39 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Kimi-VL Technical Report10 Apr
  2. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar
  3. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2025Qwen2.5-VL Technical Reportcited by 5,418
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
22 more
  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  8. 2025OpenAI GPT-5 System Cardcited by 910
  9. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  10. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  11. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  12. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  13. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  14. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  15. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  16. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  17. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 118
  18. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 112
  19. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  20. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  21. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
  22. 2026From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Visual Access Boundaries in Vision-Language Model Reasoningcited by 2
  2. 2026Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detectioncited by 2
  3. 2026Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMscited by 1
  4. 2026VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?cited by 1
  5. 2026Recompute or Reuse? Diagnosing and Mitigating Textual Shortcuts in VLM Self-Reflectioncited by 0
  6. 2026EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testingcited by 0
  7. 2026Retrieved but Not Delivered: Multimodal Memory Delivery for Long-Term Agentscited by 0
  8. 2026Certified Interface Aliases: Exact Collisions in Vision-Language Preprocessing, and When They Existcited by 0
  9. 2026InfoEdit: Probing Global Layout Reasoning in Infographic Editingcited by 0
  10. 2026AnswerMap: Faithful Spatial Interpretability of VLMs from Answer Posteriorscited by 0
  11. 2026A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversightcited by 0
  12. 2026Pay for the Fault, Not the Flow: Label-Free In-Flow Multi-Agent Workflow Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ReGround: Restoring Visual Grounding in Multi-Step Reasoning through Self-Diagnosis and Visual Re-Examination
  2. 2026Investigating Relational Reasoning in VLMs
  3. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  4. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  5. 2026Improving Visual Reasoning with Iterative Evidence Refinement
  6. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoning
  7. 2026Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions
  8. 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models