# Mind the Motions: Benchmarking Theory-of-Mind in Everyday Body Language

arXiv: https://arxiv.org/abs/2511.15887 · HTML: https://mlsanity.com/paper/2511.15887/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 72 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Builds on (most cited first, 30 of 30)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [Evaluating Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2302.02083/) (2023)
- [FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions](https://mlsanity.com/paper/2310.15421/) (2023)
- [Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker](https://mlsanity.com/paper/2306.00924/) (2023)
- [Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities](https://mlsanity.com/paper/2311.10227/) (2023)
- [MMToM-QA: Multimodal Theory of Mind Question Answering](https://mlsanity.com/paper/2401.08743/) (2024)
- [Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition](https://mlsanity.com/paper/1811.07770/) (2018)
- [MovieGraphs: Towards Understanding Human-Centric Situations from Videos](https://mlsanity.com/paper/1712.06761/) (2017)
- [iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis](https://mlsanity.com/paper/2107.00285/) (2021)
- [ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wild](https://mlsanity.com/paper/1808.09568/) (2018)
- [Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapy](https://mlsanity.com/paper/1802.01186/) (2018)
- [Social Genome: Grounded Social Reasoning Abilities of Multimodal Models](https://mlsanity.com/paper/2502.15109/) (2025)
- [AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling](https://mlsanity.com/paper/2502.15676/) (2025)
- [Do LLMs Exhibit Human-Like Reasoning? Evaluating Theory of Mind in LLMs for Open-Ended Responses](https://mlsanity.com/paper/2406.05659/) (2024)
- [Learning Triadic Belief Dynamics in Nonverbal Communication from Videos](https://mlsanity.com/paper/2104.02841/) (2021)
- [Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Task](https://mlsanity.com/paper/2409.08811/) (2024)
- [Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective](https://mlsanity.com/paper/2504.10839/) (2025)
- [VEATIC: Video-based Emotion and Affect Tracking in Context Dataset](https://mlsanity.com/paper/2309.06745/) (2023)
- [BOSS: A Benchmark for Human Belief Prediction in Object-context Scenarios](https://mlsanity.com/paper/2206.10665/) (2022)
- [Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactions](https://mlsanity.com/paper/2407.06762/) (2024)
- [BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind](https://mlsanity.com/paper/2402.07402/) (2024)
- [Bodily expressed emotion understanding through integrating Laban movement analysis](https://mlsanity.com/paper/2304.02187/) (2023)
- [Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning](https://mlsanity.com/paper/2310.19995/) (2023)
- [Probing Language Models' Gesture Understanding for Enhanced Human-AI Interaction](https://mlsanity.com/paper/2401.17858/) (2024)

## Built on by (most cited first, 1 of 1)

- [MultivationBench: A Benchmark for Multimodal Sequential Motivation Reasoning](https://mlsanity.com/paper/2607.26465/) (2026)

## Abstract (arXiv)

Our ability to interpret others' mental states through nonverbal cues (NVCs) is fundamental to our survival and social cohesion. While existing Theory of Mind (ToM) benchmarks have primarily focused on false-belief tasks and reasoning with asymmetric information, they overlook other mental states beyond belief and the rich tapestry of human nonverbal communication. We present Motion2Mind, a framework for evaluating the ToM capabilities of machines in interpreting NVCs. Leveraging an expert-curated body-language reference as a proxy knowledge base, we build Motion2Mind, a carefully curated video dataset with fine-grained nonverbal cue annotations paired with manually verified psychological interpretations. It encompasses 222 types of nonverbal cues and 397 mind states. Our evaluation reveals that current AI systems struggle significantly with NVC interpretation, exhibiting not only a substantial performance gap in Detection, as well as patterns of over-interpretation in Explanation compared to human annotators.