Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
While Multi-Modal Large Language Models (MLLMs) demonstrate impressive capabilities in general reasoning, their embodied spatial intelligence remains hampered by a "Cartesian Illusion" - a reliance on text-based probability distributions that lack grounded, 3D topological understanding. This limitation is starkly exposed in multi-agent environments, which demand more than just scene perception; they require second-order Theory of Mind (ToM). Specifically, an Agent A must be able to infer Agent B's belief about the environment, governed strictly by Agent B's physical orientation and sensory limitations. In this paper, we probe the limits of two-stage spatial inference in MLLMs through a novel audio-visual task: requiring Agent A to predict Agent B's estimation of A's relative location. To solve this, we propose an Epistemic Sensory Bottleneck module that abandons rigid, rule-based coordinate transformations. Instead, we introduce an Anchor-Based Embodied Spatial Decomposition Chain-of-Thought (CoT). This guides the MLLM through a "geometric-to-semantic" projection, forcing it to first establish B's local coordinate system and then dynamically weight visual and auditory modalities based on whether A falls within B's visual frustum. Extensive evaluations reveal that while current MLLMs fundamentally struggle with spatial symmetry and out-of-view ambiguities (establishing a rigorous zero-shot baseline of 42% accuracy), our sensory-bounded reasoning chain robustly outperforms pure egocentric and allocentric baselines. By systematically benchmarking these perceptual bottlenecks, our work exposes the current limits of MLLM spatial reasoning and establishes a foundational paradigm for epistemic, modality-aware inference in Embodied AI.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 41 references that are in the map. First posted on arXiv 18 May 2026.
Builds on
28 of 41 references.
Closest ancestors · 4
Same direction, newest first
- 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
- 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
Other references · 24
- 2023Visual Instruction Tuningcited by 7,869
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
- 2021AST: Audio Spectrogram Transformercited by 643
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
- 2020Rescaling Egocentric Visioncited by 540
- 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
- 2023An Embodied Generalist Agent in 3D Worldcited by 365
- 2017Embodied Question Answeringcited by 336
12 more
- 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
- 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
- 2018Machine Theory of Mindcited by 226
- 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
- 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
- 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
- 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 96
- 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
- 2024The 3D-PC: a benchmark for visual perspective taking in humans and machinescited by 16
- 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 15
- 2023A Review on Machine Theory of Mindcited by 9
- 2019Perspective Taking in Deep Reinforcement Learning Agentscited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs
- 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
- 2025EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks
- 2024Exploring Spatial Schema Intuitions in Large Language and Vision Models
- 2025REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories
- 2026Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models
- 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
- 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models