# Egocentric Bias in Vision-Language Models

arXiv: https://arxiv.org/abs/2602.15892 · HTML: https://mlsanity.com/paper/2602.15892/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [LLaVA-CoT: Let Vision Language Models Reason Step-by-Step](https://mlsanity.com/paper/2411.10440/) (2024)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [Improve Vision Language Model Chain-of-thought Reasoning](https://mlsanity.com/paper/2410.16198/) (2024)
- [Vision-Language Models Can Self-Improve Reasoning via Reflection](https://mlsanity.com/paper/2411.00855/) (2024)
- [Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities](https://mlsanity.com/paper/2410.17385/) (2024)
- [Can Language Models Serve as Text-Based World Simulators?](https://mlsanity.com/paper/2406.06485/) (2024)
- [Core Knowledge Deficits in Multi-Modal Language Models](https://mlsanity.com/paper/2410.10855/) (2024)
- [Exploring Perceptual Limitation of Multimodal Large Language Models](https://mlsanity.com/paper/2402.07384/) (2024)
- [Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation](https://mlsanity.com/paper/2506.21876/) (2025)
- [Vision Language Models See What You Want but not What You See](https://mlsanity.com/paper/2410.00324/) (2024)
- [Probing Mechanical Reasoning in Large Vision Language Models](https://mlsanity.com/paper/2410.00318/) (2024)
- [VIDEOP2R: Video Understanding from Perception to Reasoning](https://mlsanity.com/paper/2511.11113/) (2025)
- [Probing Perceptual Constancy in Large Vision-Language Models](https://mlsanity.com/paper/2502.10273/) (2025)
- [The Philosophical Foundations of Growing AI Like A Child](https://mlsanity.com/paper/2502.10742/) (2025)
- [Rethinking the Simulation vs. Rendering Dichotomy: No Free Lunch in Spatial World Modelling](https://mlsanity.com/paper/2510.20835/) (2025)

## Built on by (most cited first, 3 of 3)

- [DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving](https://mlsanity.com/paper/2605.23176/) (2026)
- [Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images](https://mlsanity.com/paper/2605.12413/) (2026)
- [Restrict, Don't Retrain: Inference-Time VLM Guidance for Zero-Shot Aerial Segmentation](https://mlsanity.com/paper/2609.00628/) (2026)

## Abstract (arXiv)

Visual perspective taking--inferring how the world appears from another's viewpoint--is foundational to social cognition. We introduce FlipSet, a diagnostic benchmark for Level-2 visual perspective taking (L2 VPT) in vision-language models. The task requires simulating 180-degree rotations of 2D character strings from another agent's perspective, isolating spatial transformation from 3D scene complexity. Evaluating 103 VLMs reveals systematic egocentric bias: the vast majority perform below chance, with roughly three-quarters of errors reproducing the camera viewpoint. Control experiments expose a compositional deficit--models achieve high theory-of-mind accuracy and above-chance mental rotation in isolation, yet fail catastrophically when integration is required. This dissociation indicates that current VLMs lack the mechanisms needed to bind social awareness to spatial operations, suggesting fundamental limitations in model-based spatial reasoning. FlipSet provides a cognitively grounded testbed for diagnosing perspective-taking capabilities in multimodal systems.