mlsanityData to 14 Oct 2026

EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos

arXiv 2503.22152 ↗First posted 28 Mar 2025cs.CV cs.AIcited by 12

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We introduce EgoToM, a new video question-answering benchmark that extends Theory-of-Mind (ToM) evaluation to egocentric domains. Using a causal ToM model, we generate multi-choice video QA instances for the Ego4D dataset to benchmark the ability to predict a camera wearer's goals, beliefs, and next actions. We study the performance of both humans and state of the art multimodal large language models (MLLMs) on these three interconnected inference problems. Our evaluation shows that MLLMs achieve close to human-level accuracy on inferring goals from egocentric videos. However, MLLMs (including the largest ones we tested with over 100B parameters) fall short of human performance when inferring the camera wearers' in-the-moment belief states and future actions that are most consistent with the unseen video future. We believe that our results will shape the future design of an important class of egocentric digital assistants which are equipped with a reasonable model of the user's internal mental states.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 36 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

15 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning11 Dec

Other references · 14

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  5. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  6. 2018Machine Theory of Mindcited by 226
  7. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
  8. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 115
  9. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  10. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilitiescited by 89
  11. 2020Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistencycited by 78
  12. 2020Online Bayesian Goal Inference for Boundedly-Rational Planning Agentscited by 68
2 more
  1. 2021Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of otherscited by 39
  2. 2024Towards Event-oriented Long Video Understandingcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  2. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  3. 2025Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agentscited by 8
  4. 2026MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agentscited by 5
  5. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  6. 2026A formal definition and meta-model for a machine theory of mindcited by 1
  7. 2026GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMscited by 1
  8. 2025Spot The Ball: A Benchmark for Visual Social Inferencecited by 0
  9. 2026Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Modelscited by 0
  10. 2026CoMind: Understanding Collaborative Human Activity from Multiple Minds and Viewscited by 0
  11. 2026Cross-Task Dissociation in Frontier Vision-Language Model Theory of Mindcited by 0
  12. 2026EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding
  2. 2026EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs
  3. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
  4. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos
  5. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
  6. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding
  7. 2026EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory
  8. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models