# Beyond the Visible: Benchmarking Occlusion Perception in Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2508.04059 · HTML: https://mlsanity.com/paper/2508.04059/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 26 of 26)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision](https://mlsanity.com/paper/2309.14181/) (2023)
- [Transparent Image Layer Diffusion using Latent Transparency](https://mlsanity.com/paper/2402.17113/) (2024)
- [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](https://mlsanity.com/paper/2103.12340/) (2021)
- [VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents](https://mlsanity.com/paper/2408.06327/) (2024)
- [Self-Supervised Scene De-occlusion](https://mlsanity.com/paper/2004.02788/) (2020)
- [pix2gestalt: Amodal Segmentation by Synthesizing Wholes](https://mlsanity.com/paper/2401.14398/) (2024)
- [Amodal Instance Segmentation](https://mlsanity.com/paper/1604.08202/) (2016)
- [Learning to See the Invisible: End-to-End Trainable Amodal Instance Segmentation](https://mlsanity.com/paper/1804.08864/) (2018)
- [SeGAN: Segmenting and Generating the Invisible](https://mlsanity.com/paper/1703.10239/) (2017)
- [Robust Object Detection under Occlusion with Context-Aware CompositionalNets](https://mlsanity.com/paper/2005.11643/) (2020)
- [Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models](https://mlsanity.com/paper/2502.08636/) (2025)
- [Amodal Completion and Size Constancy in Natural Scenes](https://mlsanity.com/paper/1509.08147/) (2015)
- [3D-Aware Visual Question Answering about Parts, Poses and Occlusions](https://mlsanity.com/paper/2310.17914/) (2023)
- [Using Diffusion Priors for Video Amodal Segmentation](https://mlsanity.com/paper/2412.04623/) (2024)

## Built on by (most cited first, 9 of 9)

- [MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs](https://mlsanity.com/paper/2511.14159/) (2025)
- [CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning](https://mlsanity.com/paper/2601.13304/) (2026)
- [VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs](https://mlsanity.com/paper/2512.21194/) (2025)
- [Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense](https://mlsanity.com/paper/2602.09012/) (2026)
- [DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving](https://mlsanity.com/paper/2605.23176/) (2026)
- [Spot The Ball: A Benchmark for Visual Social Inference](https://mlsanity.com/paper/2511.00261/) (2025)
- [From Street View Imagery to Street Quality Indicators: Vision Language Inference for the Suburban 15-minute City](https://mlsanity.com/paper/2608.20026/) (2026)
- [Fine-Grained Multi Image Object Hallucination Benchmark](https://mlsanity.com/paper/2608.30653/) (2026)
- [OccludeBench: Can VLMs Find What's Missing?](https://mlsanity.com/paper/2610.13790/) (2026)

## Abstract (arXiv)

Occlusion perception, a critical foundation for human-level spatial understanding, embodies the challenge of integrating visual recognition and reasoning. Though multimodal large language models (MLLMs) have demonstrated remarkable capabilities, their performance on occlusion perception remains under-explored. To address this gap, we introduce O-Bench, the first visual question answering (VQA) benchmark specifically designed for occlusion perception. Based on SA-1B, we construct 1,365 images featuring semantically coherent occlusion scenarios through a novel layered synthesis approach. Upon this foundation, we annotate 4,588 question-answer pairs in total across five tailored tasks, employing a reliable, semi-automatic workflow. Our extensive evaluation of 22 representative MLLMs against the human baseline reveals a significant performance gap between current MLLMs and humans, which, we find, cannot be sufficiently bridged by model scaling or thinking process. We further identify three typical failure patterns, including an overly conservative bias, a fragile gestalt prediction, and a struggle with quantitative tasks. We believe O-Bench can not only provide a vital evaluation tool for occlusion perception, but also inspire the development of MLLMs for better visual intelligence. Our benchmark will be made publicly available upon paper publication.