# An Exam for Active Observers

arXiv: https://arxiv.org/abs/2607.16165 · HTML: https://mlsanity.com/paper/2607.16165/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 17 Jul 2026.

## Builds on (most cited first, 24 of 24)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models](https://mlsanity.com/paper/2402.07865/) (2024)
- [CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs](https://mlsanity.com/paper/2406.18521/) (2024)
- [MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training](https://mlsanity.com/paper/2403.09611/) (2024)
- [MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs](https://mlsanity.com/paper/2502.17422/) (2025)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [Vision language models are blind: Failing to translate detailed visual features into words](https://mlsanity.com/paper/2407.06581/) (2024)
- [Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders](https://mlsanity.com/paper/2408.15998/) (2024)
- [BRAVE: Broadening the visual encoding of vision-language models](https://mlsanity.com/paper/2404.07204/) (2024)
- [Vision Language Models are Biased](https://mlsanity.com/paper/2505.23941/) (2025)
- [BabyVision: Visual Reasoning Beyond Language](https://mlsanity.com/paper/2601.06521/) (2026)
- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks](https://mlsanity.com/paper/2606.29537/) (2026)
- [cadrille: Multi-modal CAD Reconstruction with Reinforcement Learning](https://mlsanity.com/paper/2505.22914/) (2025)
- [BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD](https://mlsanity.com/paper/2605.10865/) (2026)
- [Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions](https://mlsanity.com/paper/2412.08737/) (2024)
- [MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding](https://mlsanity.com/paper/2508.15802/) (2025)
- [Blueprint-Bench: Comparing spatial intelligence of LLMs, agents and image models](https://mlsanity.com/paper/2509.25229/) (2025)
- [Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMs](https://mlsanity.com/paper/2603.20209/) (2026)
- [FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios](https://mlsanity.com/paper/2604.07413/) (2026)

## Abstract (arXiv)

Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot. Decades of psychophysics and cognitive science have argued that this active observation is essential for a wide range of tasks. Whether today's multimodal large language models (MLLMs) exercise active observation is an empirical question that current vision-language benchmarks do not answer. We introduce ActiveVision, a benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories. Tasks are designed to force repeated visual perception rather than a single static description. Frontier MLLMs collapse on ActiveVision: the highest-scoring model we evaluate, GPT-5.5 at the highest exposed reasoning-effort tier, solves only 10.6% of items and scores zero on 11 of the 17 tasks, and even Claude Fable 5, despite topping most reasoning and coding leaderboards, solves just 3.5%, far behind three human participants who average 96.1%. Furthermore, much of the gap persists even when models write and run their own vision code: such code is unreliable on realistic imagery, and catching its failures itself requires the active perception the models lack. Together, these results indicate that current MLLMs lack robust active visual observation, motivating architectures and training objectives that close the perception-reasoning loop.