Visuospatial Perspective Taking in Multimodal Language Models
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
As multimodal language models (MLMs) are increasingly used in social and collaborative settings, it is crucial to evaluate their perspective-taking abilities. Existing benchmarks largely rely on text-based vignettes or static scene understanding, leaving visuospatial perspective-taking (VPT) underexplored. We adapt two evaluation tasks from human studies: the Director Task, assessing VPT in a referential communication paradigm, and the Rotating Figure Task, probing perspective-taking across angular disparities. Across tasks, MLMs show pronounced deficits in Level 2 VPT, which requires inhibiting one's own perspective to adopt another's. These results expose critical limitations in current MLMs' ability to represent and reason about alternative perspectives, with implications for their use in collaborative contexts.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 57 references that are in the map. First posted on arXiv 4 Mar 2026.
Builds on
18 of 57 references.
Closest ancestors · 2
Same direction, newest first
- 2025Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models3 May
- 2024Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models2 Sep
Other references · 16
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Visual Instruction Tuningcited by 7,869
- 2024GPT-4o System Cardcited by 4,588
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
- 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
- 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
- 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
- 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 163
- 2024Building Machines that Learn and Think with Peoplecited by 73
- 2025International AI Safety Reportcited by 67
- 2025Re-evaluating Theory of Mind evaluation in large language modelscited by 23
4 more
- 2024The 3D-PC: a benchmark for visual perspective taking in humans and machinescited by 16
- 2024Vision Language Models See What You Want but not What You Seecited by 10
- 2024Failures in Perspective-taking of Multimodal AI Systemscited by 6
- 2024Procedural Dilemma Generation for Evaluating Moral Reasoning in Humans and Language Modelscited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Egocentric Bias in Vision-Language Models
- 2026Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
- 2026Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models
- 2025MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustness
- 2025PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Vision
- 2026See2Think: Do Multimodal Models Really Use Intermediate Visual States?
- 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
- 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought