# Visuospatial Perspective Taking in Multimodal Language Models

arXiv: https://arxiv.org/abs/2603.23510 · HTML: https://mlsanity.com/paper/2603.23510/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 57 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 18 of 18)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions](https://mlsanity.com/paper/2310.15421/) (2023)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [International AI Safety Report](https://mlsanity.com/paper/2501.17805/) (2025)
- [Re-evaluating Theory of Mind evaluation in large language models](https://mlsanity.com/paper/2502.21098/) (2025)
- [The 3D-PC: a benchmark for visual perspective taking in humans and machines](https://mlsanity.com/paper/2406.04138/) (2024)
- [Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models](https://mlsanity.com/paper/2409.12969/) (2024)
- [Vision Language Models See What You Want but not What You See](https://mlsanity.com/paper/2410.00324/) (2024)
- [Failures in Perspective-taking of Multimodal AI Systems](https://mlsanity.com/paper/2409.13929/) (2024)
- [Procedural Dilemma Generation for Evaluating Moral Reasoning in Humans and Language Models](https://mlsanity.com/paper/2404.10975/) (2024)
- [Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models](https://mlsanity.com/paper/2505.03821/) (2025)

## Built on by (most cited first, 2 of 2)

- [DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving](https://mlsanity.com/paper/2605.23176/) (2026)
- [Using profiles of cognitive capability to assess AI suitability for workplace tasks](https://mlsanity.com/paper/2608.25623/) (2026)

## Abstract (arXiv)

As multimodal language models (MLMs) are increasingly used in social and collaborative settings, it is crucial to evaluate their perspective-taking abilities. Existing benchmarks largely rely on text-based vignettes or static scene understanding, leaving visuospatial perspective-taking (VPT) underexplored. We adapt two evaluation tasks from human studies: the Director Task, assessing VPT in a referential communication paradigm, and the Rotating Figure Task, probing perspective-taking across angular disparities. Across tasks, MLMs show pronounced deficits in Level 2 VPT, which requires inhibiting one's own perspective to adopt another's. These results expose critical limitations in current MLMs' ability to represent and reason about alternative perspectives, with implications for their use in collaborative contexts.