Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Effective social interaction requires agents to translate mental state inferences into coordinated behavioral signals across verbal and nonverbal channels simultaneously. Yet existing benchmarks evaluate theory of mind (ToM) reasoning and embodied behavior in isolation, leaving unmeasured the gap between social inference and social action. We introduce MOSAIC (Multimodal Orchestration of Social Action, Inference, and Communication), a controlled benchmark in which two embodied agents interact across cooperative and competitive scenarios requiring integration of verbal statements, spatial trajectories, gaze direction, and facial expression under systematically varied ToM constraints. Evaluating 13 models, including 11 VLMs, across 200 trials per model, we find that VLMs fail to produce behaviors consistent with the expected outcomes under ToM-order constraints, and that imposing explicit ToM-order constraints produces no reliable behavioral change aligned with the specified reasoning level. Signal-level analysis reveals two sequential bottlenecks: most models cannot produce directionally coherent nonverbal signals, and even when signals are present, VLM agents fail to interpret others behaviors and react to them. PCM-LLM, included as a structured architectural reference point with an explicit ToM module, succeeds across all conditions, suggesting that explicit belief-action coupling is a sufficient ingredient for this class of tasks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 66 references that are in the map. First posted on arXiv 21 Aug 2026.
Builds on
29 of 66 references.
Closest ancestors · 2
Same direction, newest first
- 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
- 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents29 Jan
Other references · 27
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
- 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
- 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
15 more
- 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
- 2023Evaluating Large Language Models in Theory of Mind Taskscited by 222
- 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
- 2024Pixtral 12Bcited by 179
- 2023MindAgent: Emergent Gaming Interactioncited by 88
- 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 72
- 2024NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surroundingcited by 51
- 2024Werewolf Arena: A Case Study in LLM Evaluation via Social Deductioncited by 45
- 2024Position: Theory of Mind Benchmarks are Broken for Large Language Modelscited by 31
- 2024MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Modelscited by 17
- 2023Playing the Werewolf game with artificial intelligence for language understandingcited by 16
- 2025SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactionscited by 16
- 2024EgoSocialArena: Benchmarking the Social Intelligence of Large Language Models from a First-person Perspectivecited by 10
- 2025On the robustness of multimodal language model towards distractionscited by 9
- 2026Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Modelscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents
- 2025Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interaction
- 2025Doing Things with Words: Rethinking Theory of Mind Simulation in Large Language Models
- 2026From Social Reasoning to Embodied Interaction: An Agentic Framework for Social Robots
- 2026GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs
- 2025MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems
- 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction
- 2025MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mind