mlsanityData to 14 Oct 2026

Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks

arXiv 2605.18194 ↗First posted 18 May 2026cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

While Multi-Modal Large Language Models (MLLMs) demonstrate impressive capabilities in general reasoning, their embodied spatial intelligence remains hampered by a "Cartesian Illusion" - a reliance on text-based probability distributions that lack grounded, 3D topological understanding. This limitation is starkly exposed in multi-agent environments, which demand more than just scene perception; they require second-order Theory of Mind (ToM). Specifically, an Agent A must be able to infer Agent B's belief about the environment, governed strictly by Agent B's physical orientation and sensory limitations. In this paper, we probe the limits of two-stage spatial inference in MLLMs through a novel audio-visual task: requiring Agent A to predict Agent B's estimation of A's relative location. To solve this, we propose an Epistemic Sensory Bottleneck module that abandons rigid, rule-based coordinate transformations. Instead, we introduce an Anchor-Based Embodied Spatial Decomposition Chain-of-Thought (CoT). This guides the MLLM through a "geometric-to-semantic" projection, forcing it to first establish B's local coordinate system and then dynamically weight visual and auditory modalities based on whether A falls within B's visual frustum. Extensive evaluations reveal that while current MLLMs fundamentally struggle with spatial symmetry and out-of-view ambiguities (establishing a rigorous zero-shot baseline of 42% accuracy), our sensory-bounded reasoning chain robustly outperforms pure egocentric and allocentric baselines. By systematically benchmarking these perceptual bottlenecks, our work exposes the current limits of MLLM spatial reasoning and establishes a foundational paradigm for epistemic, modality-aware inference in Embodied AI.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 41 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

28 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  4. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 24

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  7. 2021AST: Audio Spectrogram Transformercited by 643
  8. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  9. 2020Rescaling Egocentric Visioncited by 540
  10. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  11. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  12. 2017Embodied Question Answeringcited by 336
12 more
  1. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  2. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  3. 2018Machine Theory of Mindcited by 226
  4. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  5. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  6. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
  7. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 96
  8. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  9. 2024The 3D-PC: a benchmark for visual perspective taking in humans and machinescited by 16
  10. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 15
  11. 2023A Review on Machine Theory of Mindcited by 9
  12. 2019Perspective Taking in Deep Reinforcement Learning Agentscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs
  2. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
  3. 2025EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks
  4. 2024Exploring Spatial Schema Intuitions in Large Language and Vision Models
  5. 2025REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories
  6. 2026Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models
  7. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
  8. 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models