mlsanityData to 7 Oct 2026

Failing to See or Failing to Know? Attributing Errors in Vision-Language Models

arXiv 2607.04683 ↗First posted 6 Jul 2026cs.CV cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) can recognize entities in clear images yet still fail when answering questions that require factual knowledge beyond what is directly observable. Prior work has either examined individual failure modes in isolation or treated incorrect answers as monolithic, binary failures. We propose a tree-structured framework that organizes failures in knowledge-intensive visual question answering into model-specific operational outcomes. Across two datasets and four VLMs, we observe consistent distributions of operational outcomes: some failures occur before entity recognition, while others persist after the relevant entity is recognized. Visual token representations are most informative for recognition-related decisions. Prompt hidden states predict answer success more effectively, although factual-access attribution remains difficult and exhibits only a weak signal. These pre-generation signals support attribution-guided routing to targeted interventions, including image repair, entity support, question rewriting, and factual evidence.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 47 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

27 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries2 Jan
  2. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models26 Nov
  3. 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
  4. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun

Other references · 23

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  6. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  7. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  8. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  9. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  10. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  11. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  12. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
11 more
  1. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  2. 2023FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Modelscited by 44
  3. 2025Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistencycited by 30
  4. 2025A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputscited by 20
  5. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  6. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Tokencited by 13
  7. 2024Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Modelscited by 9
  8. 2025Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrievalcited by 6
  9. 2025Losing the Plot: How VLM responses degrade on imperfect chartscited by 5
  10. 2025Degradation-Aware Image Enhancement via Vision-Language Classificationcited by 3
  11. 2026Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Certified Interface Aliases: Exact Collisions in Vision-Language Preprocessing, and When They Existcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
  2. 2026Seeing and Solving Are Not Enough for Vision-Language Models
  3. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  4. 2024Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models
  5. 2026Discovering Failure Modes in Vision-Language Models using RL
  6. 2026How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA
  7. 2026From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models
  8. 2026Diagnosing Visual Ignorance in Vision-Language Models