mlsanityData to 14 Oct 2026

MMToM-QA: Multimodal Theory of Mind Question Answering

arXiv 2401.08743 ↗First posted 16 Jan 2024cs.AI cs.CL cs.CVcited by 85

Not on the map. Why?

Abstract (arXiv)

Theory of Mind (ToM), the ability to understand people's mental states, is an essential ingredient for developing machines with human-level social intelligence. Recent machine learning models, particularly large language models, seem to show some aspects of ToM understanding. However, existing ToM benchmarks use unimodal datasets - either video or text. Human ToM, on the other hand, is more than video or text understanding. People can flexibly reason about another person's mind based on conceptual representations (e.g., goals, beliefs, plans) extracted from any available data. To address this, we introduce a multimodal Theory of Mind question answering (MMToM-QA) benchmark. MMToM-QA comprehensively evaluates machine ToM both on multimodal data and on different kinds of unimodal data about a person's activity in a household environment. To engineer multimodal ToM capacity, we propose a novel method, BIP-ALM (Bayesian Inverse Planning Accelerated by Language Models). BIP-ALM extracts unified representations from multimodal data and utilizes language models for scalable Bayesian inverse planning. We conducted a systematic comparison of human performance, BIP-ALM, and state-of-the-art models, including GPT-4. The experiments demonstrate that large language models and large multimodal models still lack robust ToM capacity. BIP-ALM, on the other hand, shows promising results, by leveraging the power of both model-based mental inference and language models.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 35 of its 62 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

35 of 62 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  11. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  12. 2016Cooperative Inverse Reinforcement Learningcited by 300
23 more
  1. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  2. 2018Machine Theory of Mindcited by 226
  3. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  4. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  5. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  6. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
  7. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  8. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 163
  9. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
  10. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  11. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  12. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  13. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilitiescited by 89
  14. 2018Evaluating Theory of Mind in Question Answeringcited by 64
  15. 2023Boosting Theory-of-Mind Performance in Large Language Models via Promptingcited by 64
  16. 2021MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Taskscited by 62
  17. 2021AGENT: A Benchmark for Core Psychological Reasoningcited by 42
  18. 2021Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of otherscited by 39
  19. 2023NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistantscited by 23
  20. 2021PHASE: PHysically-grounded Abstract Social Events for Machine Social Perceptioncited by 22
  21. 2022Proactive Robot Assistance via Spatio-Temporal Object Modelingcited by 18
  22. 2020StylePredict: Machine Theory of Mind for Human Driver Behavior From Trajectoriescited by 11
  23. 2023MultiVENT: Multilingual Videos of Events with Aligned Natural Textcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 43; 2026: 24.

85 papers.

Sort:

By area:

  1. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
  2. 2025Agentic Large Language Models, a surveycited by 108
  3. 2025Time-MQA: Time Series Multi-Task Question Answering with Context Enhancementcited by 82
  4. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 59
  5. 2024SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMscited by 57
  6. 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
  7. 2025Language Models use Lookbacks to Track Beliefscited by 39
  8. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  9. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 30
  10. 2025Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Modelscited by 29
  11. 2024Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planningcited by 28
  12. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
38 more
  1. 2025On Benchmarking Human-Like Intelligence in Machinescited by 25
  2. 2025Theory of Mind in Large Language Models: Assessment and Enhancementcited by 25
  3. 2025AutoToM: Scaling Model-based Mental Inference via Automated Agent Modelingcited by 23
  4. 2025Re-evaluating Theory of Mind evaluation in large language modelscited by 23
  5. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 22
  6. 2025Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human Statescited by 19
  7. 2025PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialoguescited by 17
  8. 2024GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignmentcited by 16
  9. 2025SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactionscited by 16
  10. 2025Social World Modelscited by 16
  11. 2024Can MLLMs Understand the Deep Implication Behind Chinese Images?cited by 14
  12. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  13. 2025MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mindcited by 13
  14. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
  15. 2025MimeQA: Towards Socially-Intelligent Nonverbal Foundation Modelscited by 9
  16. 2026HATS: Hardness-Aware Trajectory Synthesis for GUI Agentscited by 9
  17. 2025LVLMs are Bad at Overhearing Human Referential Communicationcited by 8
  18. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  19. 2026AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnessescited by 7
  20. 2024Understanding Epistemic Language with a Language-augmented Bayesian Theory of Mindcited by 6
  21. 2024Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mindcited by 6
  22. 2025EnigmaToM: Improve LLMs' Theory-of-Mind Reasoning Capabilities with Neural Knowledge Base of Entity Statescited by 6
  23. 2025R^3-VQA: "Read the Room" by Video Social Reasoningcited by 6
  24. 2025Language-Informed Synthesis of Rational Agent Models for Grounded Theory-of-Mind Reasoning On-The-Flycited by 6
  25. 2024MARPLE: A Benchmark for Long-Horizon Inferencecited by 5
  26. 2024NL-Eye: Abductive NLI for Imagescited by 5
  27. 2024MindForge: Empowering Embodied Agents with Theory of Mind for Lifelong Cultural Learningcited by 5
  28. 2024VAGUE: Visual Contexts Clarify Ambiguous Expressionscited by 5
  29. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Plannercited by 5
  30. 2025ToM-SSI: Evaluating Theory of Mind in Situated Social Interactionscited by 5
  31. 2025MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agentscited by 5
  32. 2025Towards Machine Theory of Mind with Large Language Model-Augmented Inverse Planningcited by 4
  33. 2026Reinforcing Human Behavior Simulation via Verbal Feedbackcited by 4
  34. 2025Towards properly implementing Theory of Mind in AI systems: An account of four misconceptionscited by 3
  35. 2026EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agentscited by 3
  36. 2025XToM: Exploring the Multilingual Theory of Mind for Large Language Modelscited by 2
  37. 2025M3FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Datasetcited by 2
  38. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Through the Theory of Mind's Eye: Reading Minds with Multimodal Video Large Language Models
  2. 2025From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models
  3. 2025Decompose-ToM: Enhancing Theory of Mind Reasoning in Large Language Models through Simulation and Task Decomposition
  4. 2025The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters
  5. 2025MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models
  6. 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
  7. 2026DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories
  8. 2024Multi-ToM: Evaluating Multilingual Theory of Mind Capabilities in Large Language Models