mlsanityData to 13 Oct 2026

How Well Can Vison-Language Models Understand Humans' Intention? An Open-ended Theory of Mind Question Evaluation Benchmark

arXiv 2503.22093 ↗First posted 28 Mar 2025cs.CV cs.AIcited by 1

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision Language Models (VLMs) have demonstrated strong reasoning capabilities in Visual Question Answering (VQA) tasks; however, their ability to perform Theory of Mind (ToM) tasks, such as inferring human intentions, beliefs, and mental states, remains underexplored. We propose an open-ended question framework to evaluate VLMs' performance across diverse categories of ToM tasks. We curated and annotated a benchmark dataset of 30 images and evaluated the performance of four VLMs of varying sizes. Our results show that the GPT-4 model outperformed all the others, with only one smaller model, GPT-4o-mini, achieving comparable performance. We observed that VLMs often struggle to infer intentions in complex scenarios such as bullying or cheating. Our findings reveal that smaller models can sometimes infer correct intentions despite relying on incorrect visual cues. The dataset is available at https://github.com/ximingwen/ToM-AAAI25-Multimodal.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 7 of its 13 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

7 of 13 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis27 Aug

Other references · 6

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2023Visual Instruction Tuningcited by 7,827
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 634
  4. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  5. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 85
  6. 2023Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoningcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?cited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating VQA in Vision Language Models using Cooperative Principles
  2. 2026Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models
  3. 2025Analyzing the Sensitivity of Vision Language Models in Visual Question Answering
  4. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  5. 2025IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ Tests
  6. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  7. 2025Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs
  8. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models