# CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Counting

arXiv: https://arxiv.org/abs/2504.15485 · HTML: https://mlsanity.com/paper/2504.15485/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 21 Apr 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 15 of 15)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://mlsanity.com/paper/2409.17146/) (2024)
- [NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples](https://mlsanity.com/paper/2410.14669/) (2024)
- [Is Temperature the Creativity Parameter of Large Language Models?](https://mlsanity.com/paper/2405.00492/) (2024)
- [CountGD: Multi-Modal Open-World Counting](https://mlsanity.com/paper/2407.04619/) (2024)
- [GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs](https://mlsanity.com/paper/2406.13246/) (2024)
- [UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling](https://mlsanity.com/paper/2408.04810/) (2024)
- [LVLM-COUNT: Enhancing the Counting Ability of Large Vision-Language Models](https://mlsanity.com/paper/2412.00686/) (2024)
- [Open-World Amodal Appearance Completion](https://mlsanity.com/paper/2411.13019/) (2024)
- [Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs](https://mlsanity.com/paper/2503.02597/) (2025)
- [Are Deep Learning Models Robust to Partial Object Occlusion in Visual Recognition Tasks?](https://mlsanity.com/paper/2409.10775/) (2024)
- [OCC-MLLM:Empowering Multimodal Large Language Model For the Understanding of Occluded Objects](https://mlsanity.com/paper/2410.01261/) (2024)

## Built on by (most cited first, 30 of 38)

- [OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models](https://mlsanity.com/paper/2506.03135/) (2025)
- [V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models](https://mlsanity.com/paper/2511.16668/) (2025)
- [TTRV: Test-Time Reinforcement Learning for Vision Language Models](https://mlsanity.com/paper/2510.06783/) (2025)
- [Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models](https://mlsanity.com/paper/2505.20728/) (2025)
- [IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting](https://mlsanity.com/paper/2512.09663/) (2025)
- [Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models](https://mlsanity.com/paper/2606.11324/) (2026)
- [Losing the Plot: How VLM responses degrade on imperfect charts](https://mlsanity.com/paper/2509.18425/) (2025)
- [RotBench: Evaluating Multimodal Large Language Models on Identifying Image Rotation](https://mlsanity.com/paper/2508.13968/) (2025)
- [Visual serial processing deficits explain divergences in human and VLM reasoning](https://mlsanity.com/paper/2509.25142/) (2025)
- [GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation](https://mlsanity.com/paper/2510.22118/) (2025)
- [Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense](https://mlsanity.com/paper/2602.09012/) (2026)
- [Can Vision-Language Models Solve the Shell Game?](https://mlsanity.com/paper/2603.08436/) (2026)
- [VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations](https://mlsanity.com/paper/2603.16506/) (2026)
- [Are VLMs Lost Between Sky and Space? LinkS2Bench for UAV-Satellite Dynamic Cross-View Spatial Intelligence](https://mlsanity.com/paper/2604.02020/) (2026)
- [What is the Right Embedding Space for Contrastive Learning in Referring Expression Counting?](https://mlsanity.com/paper/2505.22850/) (2025)
- [BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning](https://mlsanity.com/paper/2605.07394/) (2026)
- [DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving](https://mlsanity.com/paper/2605.23176/) (2026)
- [Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?](https://mlsanity.com/paper/2605.30557/) (2026)
- [Decoupling What to Count and Where to See for Referring Expression Counting](https://mlsanity.com/paper/2510.24374/) (2025)
- [Counting Through Occlusion: Framework for Open World Amodal Counting](https://mlsanity.com/paper/2511.12702/) (2025)
- [Obstruction reasoning for robotic grasping](https://mlsanity.com/paper/2511.23186/) (2025)
- [Object Counting with GPT-4o and GPT-5: A Comparative Study](https://mlsanity.com/paper/2512.03233/) (2025)
- [CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates](https://mlsanity.com/paper/2512.10342/) (2025)
- [See it to Place it: Evolving Macro Placements with Vision-Language Models](https://mlsanity.com/paper/2603.28733/) (2026)
- [PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning](https://mlsanity.com/paper/2605.19382/) (2026)
- [Spot The Ball: A Benchmark for Visual Social Inference](https://mlsanity.com/paper/2511.00261/) (2025)
- [Using street view images and visual LLMs to predict heritage values for governance support: Risks, ethics, and policy implications](https://mlsanity.com/paper/2601.06056/) (2025)
- [OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models](https://mlsanity.com/paper/2603.07786/) (2026)
- [Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoning](https://mlsanity.com/paper/2604.01764/) (2026)
- [SynMulti: Synthetic-to-Real Learning for Multimodal Video Understanding](https://mlsanity.com/paper/2604.12335/) (2026)

## Abstract (arXiv)

Recognizing and reasoning about occluded (partially or fully hidden) objects is vital to understanding visual scenes, as occlusions frequently occur in real-world environments and act as obstacles for spatial comprehension. To test models' ability to reason about multiple occluded objects, we introduce a novel task, Counting Amodally for Patterns Through Unseen REgions (CAPTURe), which requires a model to count objects arranged in a pattern by inferring how the pattern continues behind an occluder (an object which blocks parts of the scene). CAPTURe requires both recognizing visual patterns and reasoning, making it a useful testbed for evaluating vision-language models (VLMs) on whether they understand occluded patterns and possess spatial understanding skills. By requiring models to reason about occluded objects, CAPTURe also tests VLMs' ability to form world models that would allow them to fill in missing information. CAPTURe consists of two parts: (1) CAPTURe-real, with manually filtered images of real objects in patterns and (2) CAPTURe-synthetic, a controlled diagnostic with generated patterned images. We evaluate four strong VLMs (GPT-4o, Intern-VL2, Molmo, and Qwen2-VL) on CAPTURe, finding that models struggle to count on both occluded and unoccluded patterns. Crucially, we find that models perform worse with occlusion, suggesting that VLMs are also deficient in inferring unseen spatial relationships: even the strongest VLMs like GPT-4o fail to count with occlusion. In contrast, we find that humans achieve very little error on CAPTURe. We also find that providing auxiliary information of occluded object locations increases performance, underscoring that the model error comes both from an inability to handle occlusion as well as difficulty in counting in images. Code and data: https://github.com/atinpothiraj/CAPTURe