Cross-Task Dissociation in Frontier Vision-Language Model Theory of Mind
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Do frontier vision-language models present a coherent Theory-of-Mind (ToM) profile across tasks, matching the same human reference group, or does that profile fragment from one paradigm to the next? We evaluate a shared panel of nine frontier VLMs on two psychology-derived benchmarks: the Keysar Director Task (visual perspective-taking under egocentric interference) and the Frith-Happ'e animated triangles scored with the Castelli rubric (intention attribution from pure motion). On the Director Task, without chain-of-thought, the panel makes the egocentric error on 78% of trials like children rather than adults; variation is substantial across models, and reasoning rescues several models. On the triangles, the panel under-attributes intention: its ToM profile sits more than three times closer to the high-functioning-autistic-adult (HF-ASD) mean than to the typical-development-adult (TD) mean, while Goal-Directed and Random stay near TD. No model is nearest TD on both tasks; the model that looks adult-like on the Director Task falls on the HF-ASD side on the triangles, and the most TD-like model on the triangles is child-like on the Director Task. We report group-level descriptions, not diagnostic labels for any model.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 2 of its 21 references that are in the map. First posted on arXiv 31 Jul 2026.
Builds on
2 of 21 references.
Closest ancestors · 1
Same direction, newest first
Other references · 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?
- 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
- 2026Emergent Goal-Directed Attention in Large Vision-Language Models
- 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
- 2026Egocentric Bias in Vision-Language Models
- 2025Visual serial processing deficits explain divergences in human and VLM reasoning
- 2026Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
- 2026Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models