mlsanityData to 14 Oct 2026

I Spy With My Model's Eye: Visual Search as a Behavioural Test for MLLMs

arXiv 2510.19678 ↗First posted 22 Oct 2025cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) achieve strong performance on vision-language tasks, yet their visual processing is opaque. Most black-box evaluations measure task accuracy, but reveal little about underlying mechanisms. Drawing on cognitive psychology, we adapt classic visual search paradigms -- originally developed to study human perception -- to test whether MLLMs exhibit the ``pop-out'' effect, where salient visual features are detected independently of distractor set size. Using controlled experiments targeting colour, size and lighting features, we find that advanced MLLMs exhibit human-like pop-out effects in colour or size-based disjunctive (single feature) search, as well as capacity limits for conjunctive (multiple feature) search. We also find evidence to suggest that MLLMs, like humans, incorporate natural scene priors such as lighting direction into object representations. We reinforce our findings using targeted fine-tuning and mechanistic interpretability analyses. Our work shows how visual search can serve as a cognitively grounded diagnostic tool for evaluating perceptual capabilities in MLLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 40 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

17 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models22 Feb

Other references · 16

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  8. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  9. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  10. 2016Visual Question Answering: A Survey of Methods and Datasetscited by 123
  11. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  12. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problemcited by 75
4 more
  1. 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spacescited by 33
  2. 2024Towards Understanding the Robustness of LLM-based Evaluations under Perturbationscited by 10
  3. 2022Efficient Zero-shot Visual Search via Target and Context-aware Transformercited by 8
  4. 2022Target Features Affect Visual Search, A Study of Eye Fixationscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humanscited by 0
  2. 2026Human-Like Attention? A Psychophysical Comparison of Visual Search in Humans and MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Graphical Perception with Multimodal LLMs
  2. 2025The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs
  3. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
  4. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  5. 2025Visual Room 2.0: Seeing is Not Understanding for MLLMs
  6. 2026Do vision-language models search like humans? Reasoning tokens as a reaction-time analog in classic visual-search paradigms
  7. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  8. 2025Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs