# Learning Triadic Belief Dynamics in Nonverbal Communication from Videos

arXiv: https://arxiv.org/abs/2104.02841 · HTML: https://mlsanity.com/paper/2104.02841/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 59 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Deep Cosine Metric Learning for Person Re-Identification](https://mlsanity.com/paper/1812.00442/) (2018)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [Evaluating Theory of Mind in Question Answering](https://mlsanity.com/paper/1808.09352/) (2018)
- [Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense](https://mlsanity.com/paper/2004.09044/) (2020)
- [Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction](https://mlsanity.com/paper/1906.04158/) (2019)
- [LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities](https://mlsanity.com/paper/2007.15781/) (2020)
- [Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning](https://mlsanity.com/paper/1909.02144/) (2019)
- [Who is Mistaken?](https://mlsanity.com/paper/1612.01175/) (2016)
- [Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs](https://mlsanity.com/paper/2004.12248/) (2020)

## Built on by (most cited first, 17 of 17)

- [EgoTaskQA: Understanding Human Tasks in Egocentric Videos](https://mlsanity.com/paper/2210.03929/) (2022)
- [Affective Faces for Goal-Driven Dyadic Communication](https://mlsanity.com/paper/2301.10939/) (2023)
- [YouRefIt: Embodied Reference Understanding with Language and Gesture](https://mlsanity.com/paper/2109.03413/) (2021)
- [Spatial and Visual Perspective-Taking via View Rotation and Relation Reasoning for Embodied Reference Understanding](https://mlsanity.com/paper/2309.01073/) (2023)
- [Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition](https://mlsanity.com/paper/2405.12621/) (2024)
- [BOSS: A Benchmark for Human Belief Prediction in Object-context Scenarios](https://mlsanity.com/paper/2206.10665/) (2022)
- [Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene](https://mlsanity.com/paper/2108.02846/) (2021)
- [Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models](https://mlsanity.com/paper/2305.12544/) (2023)
- [Evaluating and Modeling Social Intelligence: A Comparative Study of Human and AI Capabilities](https://mlsanity.com/paper/2405.11841/) (2024)
- [From Cognition to Computation: A Comparative Review of Human Attention and Transformer Architectures](https://mlsanity.com/paper/2407.01548/) (2024)
- [Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactions](https://mlsanity.com/paper/2407.06762/) (2024)
- [The Yokai Learning Environment: Tracking Beliefs Over Space and Time](https://mlsanity.com/paper/2508.12480/) (2025)
- [AD-DINO: Attention-Dynamic DINO for Distance-Aware Embodied Reference Understanding](https://mlsanity.com/paper/2411.08451/) (2024)
- [HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer](https://mlsanity.com/paper/2305.12369/) (2023)
- [MindDial: Belief Dynamics Tracking with Theory-of-Mind Modeling for Situated Neural Dialogue Generation](https://mlsanity.com/paper/2306.15253/) (2023)
- [Mind the Motions: Benchmarking Theory-of-Mind in Everyday Body Language](https://mlsanity.com/paper/2511.15887/) (2025)
- [IntentQA: Intent Question Answering in Videos by Cognitive Context Reasoning](https://mlsanity.com/paper/2608.23330/) (2026)

## Abstract (arXiv)

Humans possess a unique social cognition capability; nonverbal communication can convey rich social information among agents. In contrast, such crucial social characteristics are mostly missing in the existing scene understanding literature. In this paper, we incorporate different nonverbal communication cues (e.g., gaze, human poses, and gestures) to represent, model, learn, and infer agents' mental states from pure visual inputs. Crucially, such a mental representation takes the agent's belief into account so that it represents what the true world state is and infers the beliefs in each agent's mental state, which may differ from the true world states. By aggregating different beliefs and true world states, our model essentially forms "five minds" during the interactions between two agents. This "five minds" model differs from prior works that infer beliefs in an infinite recursion; instead, agents' beliefs are converged into a "common mind". Based on this representation, we further devise a hierarchical energy-based model that jointly tracks and predicts all five minds. From this new perspective, a social event is interpreted by a series of nonverbal communication and belief dynamics, which transcends the classic keyframe video summary. In the experiments, we demonstrate that using such a social account provides a better video summary on videos with rich social interactions compared with state-of-the-art keyframe video summary methods.