mlsanityData to 14 Oct 2026

OccludeBench: Can VLMs Find What's Missing?

arXiv 2610.13790 ↗First posted 9 Oct 2026cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

How well can vision-language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 46 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

37 of 46 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models20 Feb
  2. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs20 Oct
  3. 2025Beyond the Visible: Benchmarking Occlusion Perception in Multimodal Large Language Models6 Aug
  4. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models21 Apr
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2025Qwen3-VL Technical Reportcited by 2,793
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
18 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  2. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  4. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  5. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  6. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  7. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  8. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  9. 2022Visual Spatial Reasoningcited by 293
  10. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 160
  11. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 153
  12. 2019Shadow Removal via Shadow Image Decompositioncited by 98
  13. 2023Amodal Ground Truth and Completion in the Wildcited by 45
  14. 2025CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Countingcited by 38
  15. 2024Open-World Amodal Appearance Completioncited by 19
  16. 2024Removing Reflections from RAW Photoscited by 10
  17. 2024Unveiling Deep Shadows: A Survey and Benchmark on Image and Video Shadow Detection, Removal, and Generation in the Deep Learning Eracited by 7
  18. 2025From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answeringcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts
  2. 2026How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures
  3. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  4. 2026VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?
  5. 2026VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs
  6. 2026Composition, Not Conversation: VLMs Lose the Scene, Not the Thread
  7. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
  8. 2025Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations