mlsanityData to 14 Oct 2026

Mind the Motions: Benchmarking Theory-of-Mind in Everyday Body Language

arXiv 2511.15887 ↗First posted 19 Nov 2025cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Our ability to interpret others' mental states through nonverbal cues (NVCs) is fundamental to our survival and social cohesion. While existing Theory of Mind (ToM) benchmarks have primarily focused on false-belief tasks and reasoning with asymmetric information, they overlook other mental states beyond belief and the rich tapestry of human nonverbal communication. We present Motion2Mind, a framework for evaluating the ToM capabilities of machines in interpreting NVCs. Leveraging an expert-curated body-language reference as a proxy knowledge base, we build Motion2Mind, a carefully curated video dataset with fine-grained nonverbal cue annotations paired with manually verified psychological interpretations. It encompasses 222 types of nonverbal cues and 397 mind states. Our evaluation reveals that current AI systems struggle significantly with NVC interpretation, exhibiting not only a substantial performance gap in Detection, as well as patterns of over-interpretation in Explanation compared to human annotators.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 72 references that are in the map. First posted on arXiv 19 Nov 2025.

Builds on

30 of 72 references.

References · 30

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  6. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 583
  7. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  8. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 222
  9. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  10. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  11. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilitiescited by 89
  12. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 85
18 more
  1. 2018Aff-Wild2: Extending the Aff-Wild Database for Affect Recognitioncited by 81
  2. 2017MovieGraphs: Towards Understanding Human-Centric Situations from Videoscited by 67
  3. 2021iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysiscited by 63
  4. 2018ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wildcited by 34
  5. 2018Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapycited by 29
  6. 2025Social Genome: Grounded Social Reasoning Abilities of Multimodal Modelscited by 26
  7. 2025AutoToM: Scaling Model-based Mental Inference via Automated Agent Modelingcited by 23
  8. 2024Do LLMs Exhibit Human-Like Reasoning? Evaluating Theory of Mind in LLMs for Open-Ended Responsescited by 22
  9. 2021Learning Triadic Belief Dynamics in Nonverbal Communication from Videoscited by 17
  10. 2024Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Taskcited by 17
  11. 2025Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspectivecited by 14
  12. 2023VEATIC: Video-based Emotion and Affect Tracking in Context Datasetcited by 12
  13. 2022BOSS: A Benchmark for Human Belief Prediction in Object-context Scenarioscited by 8
  14. 2024Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactionscited by 6
  15. 2024BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mindcited by 5
  16. 2023Bodily expressed emotion understanding through integrating Laban movement analysiscited by 3
  17. 2023Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoningcited by 3
  18. 2024Probing Language Models' Gesture Understanding for Enhanced Human-AI Interactioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026MultivationBench: A Benchmark for Multimodal Sequential Motivation Reasoningcited by 1