mlsanityData to 7 Oct 2026

Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models

arXiv 2409.12969 ↗First posted 2 Sep 2024cs.CL cs.CV cs.LGcited by 15

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Visual perspective-taking (VPT), the ability to understand the viewpoint of another person, enables individuals to anticipate the actions of other people. For instance, a driver can avoid accidents by assessing what pedestrians see. Humans typically develop this skill in early childhood, but it remains unclear whether the recently emerging Vision Language Models (VLMs) possess such capability. Furthermore, as these models are increasingly deployed in the real world, understanding how they perform nuanced tasks like VPT becomes essential. In this paper, we introduce two manually curated datasets, Isle-Bricks and Isle-Dots for testing VPT skills, and we use it to evaluate 12 commonly used VLMs. Across all models, we observe a significant performance drop when perspective-taking is required. Additionally, we find performance in object detection tasks is poorly correlated with performance on VPT tasks, suggesting that the existing benchmarks might not be sufficient to understand this problem. The code and the dataset will be available at https://sites.google.com/view/perspective-taking

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 46 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

24 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Robotic Control via Embodied Chain-of-Thought Reasoning11 Jul
  2. 2024CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environments22 Mar
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 21

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2023Large Language Modelscited by 774
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  6. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  7. 2022Visual Classification via Description from Large Language Modelscited by 331
  8. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  9. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  10. 2024An Introduction to Vision-Language Modelingcited by 175
  11. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  12. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
9 more
  1. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  2. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  3. 2024The Neglected Tails in Vision-Language Modelscited by 63
  4. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  5. 2023Fool Your (Vision and) Language Model With Embarrassingly Simple Permutationscited by 24
  6. 2022Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalizationcited by 20
  7. 2024The 3D-PC: a benchmark for visual perspective taking in humans and machinescited by 16
  8. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answeringcited by 14
  9. 2024D'ej`a Vu Memorization in Vision-Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 41
  2. 2025Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspectivecited by 7
  3. 2026Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiationcited by 5
  4. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  5. 2026Visuospatial Perspective Taking in Multimodal Language Modelscited by 2
  6. 2025PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Visioncited by 1
  7. 2026Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Modelscited by 1
  8. 2025Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worldscited by 0
  9. 2025Who Sees What? Structured Thought-Action Sequences for Epistemic Reasoning in LLMscited by 0
  10. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0
  11. 2026Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Imagescited by 0
  12. 2026OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignmentcited by 0
3 more
  1. 2026SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Modelscited by 0
  2. 2026WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoningcited by 0
  3. 2026SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models
  2. 2026Egocentric Bias in Vision-Language Models
  3. 2025VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues
  4. 2025Vision-Language Models Mistake Head Orientation for Gaze Direction: Nonverbal Conversation Cues
  5. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models
  6. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  7. 2025Visual Language Models show widespread visual deficits on neuropsychological tests
  8. 2026Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models