# EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos

arXiv: https://arxiv.org/abs/2503.22152 · HTML: https://mlsanity.com/paper/2503.22152/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [EgoTaskQA: Understanding Human Tasks in Egocentric Videos](https://mlsanity.com/paper/2210.03929/) (2022)
- [Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker](https://mlsanity.com/paper/2306.00924/) (2023)
- [Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities](https://mlsanity.com/paper/2311.10227/) (2023)
- [Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency](https://mlsanity.com/paper/2006.16736/) (2020)
- [EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning](https://mlsanity.com/paper/2312.06722/) (2023)
- [Online Bayesian Goal Inference for Boundedly-Rational Planning Agents](https://mlsanity.com/paper/2006.07532/) (2020)
- [Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others](https://mlsanity.com/paper/2102.11938/) (2021)
- [Towards Event-oriented Long Video Understanding](https://mlsanity.com/paper/2406.14129/) (2024)

## Built on by (most cited first, 12 of 12)

- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning](https://mlsanity.com/paper/2506.13654/) (2025)
- [Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents](https://mlsanity.com/paper/2510.22443/) (2025)
- [MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents](https://mlsanity.com/paper/2603.09827/) (2026)
- [EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning](https://mlsanity.com/paper/2511.18242/) (2025)
- [A formal definition and meta-model for a machine theory of mind](https://mlsanity.com/paper/2606.03471/) (2026)
- [GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs](https://mlsanity.com/paper/2606.04184/) (2026)
- [Spot The Ball: A Benchmark for Visual Social Inference](https://mlsanity.com/paper/2511.00261/) (2025)
- [Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models](https://mlsanity.com/paper/2603.24484/) (2026)
- [CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views](https://mlsanity.com/paper/2607.06691/) (2026)
- [Cross-Task Dissociation in Frontier Vision-Language Model Theory of Mind](https://mlsanity.com/paper/2608.00261/) (2026)
- [EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions](https://mlsanity.com/paper/2610.18499/) (2026)

## Abstract (arXiv)

We introduce EgoToM, a new video question-answering benchmark that extends Theory-of-Mind (ToM) evaluation to egocentric domains. Using a causal ToM model, we generate multi-choice video QA instances for the Ego4D dataset to benchmark the ability to predict a camera wearer's goals, beliefs, and next actions. We study the performance of both humans and state of the art multimodal large language models (MLLMs) on these three interconnected inference problems. Our evaluation shows that MLLMs achieve close to human-level accuracy on inferring goals from egocentric videos. However, MLLMs (including the largest ones we tested with over 100B parameters) fall short of human performance when inferring the camera wearers' in-the-moment belief states and future actions that are most consistent with the unseen video future. We believe that our results will shape the future design of an important class of egocentric digital assistants which are equipped with a reasonable model of the user's internal mental states.