mlsanityData to 7 Oct 2026

Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?

arXiv 2512.17394 ↗First posted 19 Dec 2025cs.CL cs.CV cs.CYcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Theory of Mind (ToM) - the ability to attribute beliefs and intents to others - is fundamental for social intelligence, yet Vision-Language Model (VLM) evaluations remain largely Western-centric. In this work, we introduce CulturalToM-VQA, a benchmark of 5,095 visually situated ToM probes across diverse cultural contexts, rituals, and social norms. Constructed through a frontier proprietary MLLM, human-verified pipeline, the dataset spans a taxonomy of six ToM tasks and four complexity levels. We benchmark 10 VLMs (2023-2025) and observe a significant performance leap: while earlier models struggle, frontier models achieve high accuracy (>93%). However, significant limitations persist: models struggle with false belief reasoning (19-83% accuracy) and show high regional variance (20-30% gaps). Crucially, we find that SOTA models exhibit social desirability bias - systematically favoring semantically positive answer choices over negative ones. Ablation experiments reveal that some frontier models rely heavily on parametric social priors, frequently defaulting to safety-aligned predictions. Furthermore, while Chain-of-Thought prompting aids older models, it yields minimal gains for newer ones. Overall, our work provides a testbed for cross-cultural social reasoning, underscoring that despite architectural gains, achieving robust, visually grounded understanding remains an open challenge.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 41 of its 68 references that are in the map. First posted on arXiv 19 Dec 2025.

Builds on

41 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025R^3-VQA: "Read the Room" by Video Social Reasoning7 May
  2. 2024OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models8 Feb
  3. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities16 Nov
  4. 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models25 Oct
  5. 2023Understanding Social Reasoning in Language Models with Language Models21 Jun
  6. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models24 May

Other references · 35

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  6. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  7. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 217
  8. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  9. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  10. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
  11. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  12. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 100
23 more
  1. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  2. 2020Context Based Emotion Recognition using EMOTIC Datasetcited by 73
  3. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 70
  4. 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
  5. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  6. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countriescited by 40
  7. 2024WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisinescited by 39
  8. 2024FindingEmo: An Image Dataset for Emotion Recognition in the Wildcited by 20
  9. 2025Evaluating Vision-Language Models for Emotion Recognitioncited by 19
  10. 2025SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactionscited by 16
  11. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Modelscited by 14
  12. 2024Visual Prompting in LLMs for Enhancing Emotion Recognitioncited by 12
  13. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  14. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  15. 2024Contextual Emotion Recognition using Large Vision Language Modelscited by 8
  16. 2024Failures in Perspective-taking of Multimodal AI Systemscited by 6
  17. 2024Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Costcited by 5
  18. 2024Multi-ToM: Evaluating Multilingual Theory of Mind Capabilities in Large Language Modelscited by 5
  19. 2025From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Modelscited by 5
  20. 2024AToM-Bot: Embodied Fulfillment of Unspoken Human Needs with Affective Theory of Mindcited by 4
  21. 2025VIBE: Can a VLM Read the Room?cited by 4
  22. 2024Clean Evaluations on Contaminated Visual Language Modelscited by 3
  23. 2025How Well Can Vison-Language Models Understand Humans' Intention? An Open-ended Theory of Mind Question Evaluation Benchmarkcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025XToM: Exploring the Multilingual Theory of Mind for Large Language Models
  2. 2025Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
  3. 2024Probing the Robustness of Theory of Mind in Large Language Models
  4. 2024Language Models Represent Beliefs of Self and Others
  5. 2026Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models
  6. 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models
  7. 2026CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs
  8. 2024Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Models