# MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mind

arXiv: https://arxiv.org/abs/2507.04415 · HTML: https://mlsanity.com/paper/2507.04415/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 6 Jul 2025.

## Builds on (most cited first, 9 of 9)

- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [MMToM-QA: Multimodal Theory of Mind Question Answering](https://mlsanity.com/paper/2401.08743/) (2024)
- [Social Genome: Grounded Social Reasoning Abilities of Multimodal Models](https://mlsanity.com/paper/2502.15109/) (2025)
- [Theory of Mind in Large Language Models: Assessment and Enhancement](https://mlsanity.com/paper/2505.00026/) (2025)
- [Deep Interpretable Models of Theory of Mind](https://mlsanity.com/paper/2104.02938/) (2021)
- [Long Story Short: Story-level Video Understanding from 20K Short Films](https://mlsanity.com/paper/2406.10221/) (2024)
- [DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding](https://mlsanity.com/paper/2310.18359/) (2023)

## Built on by (most cited first, 13 of 13)

- [MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents](https://mlsanity.com/paper/2511.23055/) (2025)
- [Cognitive Chain-of-Thought (CoCoT): Structured Multimodal Reasoning about Social Situations](https://mlsanity.com/paper/2507.20409/) (2025)
- [A formal definition and meta-model for a machine theory of mind](https://mlsanity.com/paper/2606.03471/) (2026)
- [GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs](https://mlsanity.com/paper/2606.04184/) (2026)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)
- [Understanding Mental States to Guide Social Influence in Multi-Person Group Dialogue](https://mlsanity.com/paper/2601.13687/) (2026)
- [MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention](https://mlsanity.com/paper/2606.01063/) (2026)
- [CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views](https://mlsanity.com/paper/2607.06691/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [MoCA: Implicit Social Context Analysis](https://mlsanity.com/paper/2608.05825/) (2026)
- [SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos](https://mlsanity.com/paper/2608.30716/) (2026)
- [MindWorldBench: Evaluating Mental-State-to-Behavior Reasoning in Image-to-Video Generation](https://mlsanity.com/paper/2609.39147/) (2026)
- [PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios](https://mlsanity.com/paper/2610.20548/) (2026)

## Abstract (arXiv)

Understanding Theory of Mind is essential for building socially intelligent multimodal agents capable of perceiving and interpreting human behavior. We introduce MoMentS (Multimodal Mental States), a comprehensive benchmark designed to assess the ToM capabilities of multimodal large language models (LLMs) through realistic, narrative-rich scenarios presented in short films. MoMentS includes over 2,300 multiple-choice questions spanning seven distinct ToM categories. The benchmark features long video context windows and realistic social interactions that provide deeper insight into characters' mental states. We evaluate several MLLMs and find that although vision generally improves performance, models still struggle to integrate it effectively. For audio, models that process dialogues as audio do not consistently outperform transcript-based inputs. Our findings highlight the need to improve multimodal integration and point to open challenges that must be addressed to advance AI's social understanding.