# Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey

arXiv: https://arxiv.org/abs/2207.10574 · HTML: https://mlsanity.com/paper/2207.10574/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 216 references that are in the map.
- First posted on arXiv 20 Jul 2022.

## Builds on (most cited first, 25 of 25)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [ConvoKit: A Toolkit for the Analysis of Conversations](https://mlsanity.com/paper/2005.04246/) (2020)
- [Locally Private Graph Neural Networks](https://mlsanity.com/paper/2006.05535/) (2020)
- [Context-Aware Personality Inference in Dyadic Scenarios: Introducing the UDIVA Dataset](https://mlsanity.com/paper/2012.14259/) (2020)
- [Recent Trends in Deep Learning Based Personality Detection](https://mlsanity.com/paper/1908.03628/) (2019)
- [Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour](https://mlsanity.com/paper/1801.06055/) (2018)
- [First Impressions: A Survey on Vision-Based Apparent Personality Trait Analysis](https://mlsanity.com/paper/1804.08046/) (2018)
- [Unsupervised Human Action Recognition with Skeletal Graph Laplacian and Self-Supervised Viewpoints Invariance](https://mlsanity.com/paper/2204.10312/) (2022)
- [With Whom Do I Interact? Detecting Social Interactions in Egocentric Photo-streams](https://mlsanity.com/paper/1605.04129/) (2016)
- [Analyzing Verbal and Nonverbal Features for Predicting Group Performance](https://mlsanity.com/paper/1907.01369/) (2019)
- [Subspace Clustering for Action Recognition with Covariance Representations and Temporal Pruning](https://mlsanity.com/paper/2006.11812/) (2020)
- [Predicting Group Cohesiveness in Images](https://mlsanity.com/paper/1812.11771/) (2018)
- [Emergent Leadership Detection Across Datasets](https://mlsanity.com/paper/1905.02058/) (2019)
- [Detecting F-formations & Roles in Crowded Social Scenes with Wearables: Combining Proxemics & Dynamics using LSTMs](https://mlsanity.com/paper/1911.07279/) (2019)
- [Vyaktitv: A Multimodal Peer-to-Peer Hindi Conversations based Dataset for Personality Assessment](https://mlsanity.com/paper/2008.13769/) (2020)
- [Unsupervised Multimodal Language Representations using Convolutional Autoencoders](https://mlsanity.com/paper/2110.03007/) (2021)
- [Finding Dory in the Crowd: Detecting Social Interactions using Multi-Modal Mobile Sensing](https://mlsanity.com/paper/1809.00947/) (2018)
- [Social Relation Recognition in Egocentric Photostreams](https://mlsanity.com/paper/1905.04734/) (2019)
- [Detecting Personality and Emotion Traits in Crowds from Video Sequences](https://mlsanity.com/paper/2104.12927/) (2021)

## Built on by (most cited first, 6 of 6)

- [Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes](https://mlsanity.com/paper/2410.22177/) (2024)
- [Upper-Body Pose-based Gaze Estimation for Privacy-Preserving 3D Gaze Target Detection](https://mlsanity.com/paper/2409.17886/) (2024)
- [AL-GTD: Deep Active Learning for Gaze Target Detection](https://mlsanity.com/paper/2409.18561/) (2024)
- [Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning](https://mlsanity.com/paper/2606.29334/) (2026)
- [SICAGE: Speaker-Independent Culture-Aware Gesture Generation using TED4C-L Dataset](https://mlsanity.com/paper/2606.30001/) (2026)
- [Human-Inspired Social Engagement Analysis via Interpretable Mutual Visual Attention](https://mlsanity.com/paper/2608.24580/) (2026)

## Abstract (arXiv)

Automated co-located human-human interaction analysis has been addressed by the use of nonverbal communication as measurable evidence of social and psychological phenomena. We survey the computing studies (since 2010) detecting phenomena related to social traits (e.g., leadership, dominance, personality traits), social roles/relations, and interaction dynamics (e.g., group cohesion, engagement, rapport). Our target is to identify the nonverbal cues and computational methodologies resulting in effective performance. This survey differs from its counterparts by involving the widest spectrum of social phenomena and interaction settings (free-standing conversations, meetings, dyads, and crowds). We also present a comprehensive summary of the related datasets and outline future research directions which are regarding the implementation of artificial intelligence, dataset curation, and privacy-preserving interaction analysis. Some major observations are: the most often used nonverbal cue, computational method, interaction environment, and sensing approach are speaking activity, support vector machines, and meetings composed of 3-4 persons equipped with microphones and cameras, respectively; multimodal features are prominently performing better; deep learning architectures showed improved performance in overall, but there exist many phenomena whose detection has never been implemented through deep models. We also identified several limitations such as the lack of scalable benchmarks, annotation reliability tests, cross-dataset experiments, and explainability analysis.