# Towards Social AI: A Survey on Understanding Social Interactions

arXiv: https://arxiv.org/abs/2409.15316 · HTML: https://mlsanity.com/paper/2409.15316/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2024.

## Built on by (most cited first, 16 of 16)

- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [Towards Online Multi-Modal Social Interaction Understanding](https://mlsanity.com/paper/2503.19851/) (2025)
- [SocialGesture: Delving into Multi-person Gesture Understanding](https://mlsanity.com/paper/2504.02244/) (2025)
- [Multi-speaker Attention Alignment for Multimodal Social Interaction](https://mlsanity.com/paper/2511.17952/) (2025)
- [VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs](https://mlsanity.com/paper/2511.20272/) (2025)
- [MotiveBench: How Far Are We From Human-Like Motivational Reasoning in Large Language Models?](https://mlsanity.com/paper/2506.13065/) (2025)
- [ChatThero: An LLM-Supported Chatbot for Behavior Change and Therapeutic Support in Addiction Recovery](https://mlsanity.com/paper/2508.20996/) (2025)
- [Omni-MMSI: Toward Identity-attributed Social Interaction Understanding](https://mlsanity.com/paper/2604.00267/) (2026)
- [SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution](https://mlsanity.com/paper/2604.18982/) (2026)
- [SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generation](https://mlsanity.com/paper/2605.10079/) (2026)
- [ChatCLIDS: Simulating Persuasive AI Dialogues to Promote Closed-Loop Insulin Adoption in Type 1 Diabetes Care](https://mlsanity.com/paper/2509.00891/) (2025)
- [GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions](https://mlsanity.com/paper/2605.15764/) (2026)
- [Evaluating Large Language Models in a Complex Hidden Role Game](https://mlsanity.com/paper/2605.22826/) (2026)
- [MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings](https://mlsanity.com/paper/2607.19235/) (2026)
- [Collaborative Participatory Research with LLM Agents in South Asia: An Empirically-Grounded Methodological Initiative and Agenda from Field Evidence in Sri Lanka](https://mlsanity.com/paper/2411.08294/) (2024)
- [CSGaze: Context-aware Social Gaze Prediction](https://mlsanity.com/paper/2511.05955/) (2025)

## Abstract (arXiv)

Social interactions form the foundation of human societies. Artificial intelligence has made significant progress in certain areas, but enabling machines to seamlessly understand social interactions remains an open challenge. It is important to address this gap by endowing machines with social capabilities. We identify three key capabilities needed for effective social understanding: 1) understanding multimodal social cues, 2) understanding multi-party dynamics, and 3) understanding beliefs. Building upon these foundations, we classify and review existing machine learning works on social understanding from the perspectives of verbal, non-verbal, and multimodal social cues. The verbal branch focuses on understanding linguistic signals such as speaker intent, dialogue sentiment, and commonsense reasoning. The non-verbal branch addresses techniques for perceiving social meaning from visual behaviors such as body gestures, gaze patterns, and facial expressions. The multimodal branch covers approaches that integrate verbal and non-verbal multimodal cues to holistically interpret social interactions such as recognizing emotions, conversational dynamics, and social situations. By reviewing the scope and limitations of current approaches and benchmarks, we aim to clarify the development trajectory and illuminate the path towards more comprehensive intelligence for social understanding. We hope this survey will spur further research interest and insights into this area.