mlsanityData to 7 Oct 2026

Vision Language Models See What You Want but not What You See

arXiv 2410.00324 ↗First posted 1 Oct 2024cs.AIcited by 10

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Knowing others' intentions and taking others' perspectives are two core components of human intelligence that are considered to be instantiations of theory-of-mind. Infiltrating machines with these abilities is an important step towards building human-level artificial intelligence. Here, to investigate intentionality understanding and level-2 perspective-taking in Vision Language Models (VLMs), we constructed the IntentBench and PerspectBench, which together contains over 300 cognitive experiments grounded in real-world scenarios and classic cognitive tasks. We found VLMs achieving high performance on intentionality understanding but low performance on level-2 perspective-taking. This suggests a potential dissociation between simulation-based and theory-based theory-of-mind abilities in VLMs, highlighting the concern that they are not capable of using model-based reasoning to infer others' mental states.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 112 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

35 of 112 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024CogVLM2: Visual Language Models for Image and Video Understanding29 Aug
  2. 2024Vision language models are blind: Failing to translate detailed visual features into words9 Jul
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
19 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  4. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  5. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  6. 2023Instruction Tuning with GPT-4cited by 676
  7. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  8. 2018Human Action Recognition and Prediction: A Surveycited by 226
  9. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  10. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  11. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  12. 2016Going Deeper into Action Recognition: A Surveycited by 55
  13. 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
  14. 2023From task structures to world models: What do LLMs know?cited by 46
  15. 2024COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuningcited by 45
  16. 2022Consistent-Teacher: Towards Reducing Inconsistent Pseudo-targets in Semi-supervised Object Detectioncited by 43
  17. 2021AGENT: A Benchmark for Core Psychological Reasoningcited by 42
  18. 2021More than Encoder: Introducing Transformer Decoder to Upsamplecited by 26
  19. 2023Diverse Cotraining Makes Strong Semi-Supervised Segmentorcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025The Philosophical Foundations of Growing AI Like A Childcited by 6
  2. 2025Video Models Start to Solve Chess, Maze, Sudoku, Mental Rotation, and Raven' Matricescited by 4
  3. 2026Vision Language Models Cannot Reason About Physical Transformationcited by 4
  4. 2026Egocentric Bias in Vision-Language Modelscited by 3
  5. 2026Visuospatial Perspective Taking in Multimodal Language Modelscited by 2
  6. 2025Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questionscited by 1
  7. 2025Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worldscited by 0
  8. 2026Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Imagescited by 0
  9. 2026Cross-Task Dissociation in Frontier Vision-Language Model Theory of Mindcited by 0
  10. 2026MindTopo: Can Foundation Models Reason in Topological Space?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities
  2. 2025How Well Can Vison-Language Models Understand Humans' Intention? An Open-ended Theory of Mind Question Evaluation Benchmark
  3. 2024Analyzing the Roles of Language and Vision in Learning from Limited Data
  4. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation
  5. 2025Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?
  6. 2023Visual cognition in multimodal large language models
  7. 2025MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents
  8. 2022Mind's Eye: Grounded Language Model Reasoning through Simulation