# Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models

arXiv: https://arxiv.org/abs/2601.16378 · HTML: https://mlsanity.com/paper/2601.16378/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 14 of 14)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark](https://mlsanity.com/paper/2412.07825/) (2024)
- [Perception Tokens Enhance Visual Reasoning in Multimodal Language Models](https://mlsanity.com/paper/2412.03548/) (2024)
- [Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities](https://mlsanity.com/paper/2410.17385/) (2024)
- [Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models](https://mlsanity.com/paper/2412.18605/) (2024)
- [Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation](https://mlsanity.com/paper/2504.17207/) (2025)
- [SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation](https://mlsanity.com/paper/2412.12693/) (2024)
- [Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models](https://mlsanity.com/paper/2409.12969/) (2024)
- [Failures in Perspective-taking of Multimodal AI Systems](https://mlsanity.com/paper/2409.13929/) (2024)
- [Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models](https://mlsanity.com/paper/2505.03821/) (2025)

## Built on by (most cited first, 1 of 1)

- [OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignment](https://mlsanity.com/paper/2607.17657/) (2026)

## Abstract (arXiv)

Multimodal language models (MLMs) perform well on semantic vision-language tasks but fail at spatial reasoning that requires adopting another agent's visual perspective. These errors reflect a persistent egocentric bias and raise questions about whether current models support allocentric reasoning. Inspired by human spatial cognition, we introduce perspective tokens, specialized embeddings that encode orientation through either (1) embodied body-keypoint cues or (2) abstract representations supporting mental rotation. Integrating these tokens into LLaVA-1.5-13B yields performance on level-2 visual perspective-taking tasks. Across synthetic and naturalistic benchmarks (Isle Bricks V2, COCO, 3DSRBench), perspective tokens improve accuracy, with rotation-based tokens generalizing to non-human reference agents. Representational analyses reveal that fine-tuning enhances latent orientation sensitivity already present in the base model, suggesting that MLMs contain precursors of allocentric reasoning but lack appropriate internal structure. Overall, embedding cognitively grounded spatial structure directly into token space provides a lightweight, model-agnostic mechanism for perspective-taking and more human-like spatial reasoning.