# Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour

arXiv: https://arxiv.org/abs/1801.06055 · HTML: https://mlsanity.com/paper/1801.06055/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2018.

## Built on by (most cited first, 24 of 24)

- [Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation](https://mlsanity.com/paper/2207.12817/) (2022)
- [Data Augmentation for Human Behavior Analysis in Multi-Person Conversations](https://mlsanity.com/paper/2308.01526/) (2023)
- [An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile Devices](https://mlsanity.com/paper/2307.00122/) (2023)
- [ConfLab: A Data Collection Concept, Dataset, and Benchmark for Machine Analysis of Free-Standing Social Interactions in the Wild](https://mlsanity.com/paper/2205.05177/) (2022)
- [Multi-speaker Attention Alignment for Multimodal Social Interaction](https://mlsanity.com/paper/2511.17952/) (2025)
- [Backchannel Detection and Agreement Estimation from Video with Transformer Networks](https://mlsanity.com/paper/2306.01656/) (2023)
- [Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey](https://mlsanity.com/paper/2207.10574/) (2022)
- [Gaze-enhanced Crossmodal Embeddings for Emotion Recognition](https://mlsanity.com/paper/2205.00129/) (2022)
- [DAT: Dialogue-Aware Transformer with Modality-Group Fusion for Human Engagement Estimation](https://mlsanity.com/paper/2410.08470/) (2024)
- [Multimodal Vision Transformers with Forced Attention for Behavior Analysis](https://mlsanity.com/paper/2212.03968/) (2022)
- [MultiMediate'23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions](https://mlsanity.com/paper/2308.08256/) (2023)
- [MONAH: Multi-Modal Narratives for Humans to analyze conversations](https://mlsanity.com/paper/2101.07339/) (2021)
- [HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer](https://mlsanity.com/paper/2305.12369/) (2023)
- [MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings](https://mlsanity.com/paper/2309.10765/) (2023)
- [Generalizable Engagement Estimation in Conversation via Domain Prompting and Parallel Attention](https://mlsanity.com/paper/2508.14448/) (2025)
- [A survey of neural models for the automatic analysis of conversation: Towards a better integration of the social sciences](https://mlsanity.com/paper/2203.16891/) (2022)
- [Investigating Role of Big Five Personality Traits in Audio-Visual Rapport Estimation](https://mlsanity.com/paper/2410.11861/) (2024)
- [CMIS-Net: A Cascaded Multi-Scale Individual Standardization Network for Backchannel Agreement Estimation](https://mlsanity.com/paper/2510.17855/) (2025)
- [Inclusive AI for Group Interactions: Predicting Gaze-Direction Behaviors in People with Intellectual and Developmental Disabilities](https://mlsanity.com/paper/2603.14460/) (2026)
- [B-MoE: A Body-Part-Aware Mixture-of-Experts "All Parts Matter" Approach to Micro-Action Recognition](https://mlsanity.com/paper/2603.24245/) (2026)
- [GroupAffect-4: A Multimodal Dataset of Four-Person Collaborative Interaction](https://mlsanity.com/paper/2605.19765/) (2026)
- [Memory-Driven Self-Disclosure and Relational Turning Points: A Longitudinal Multimodal Study of Human-AI Interaction](https://mlsanity.com/paper/2607.14593/) (2026)
- [Towards Actionable Surgical Team Dynamics: from Teamwork to Counterfactual Annotations](https://mlsanity.com/paper/2608.23344/) (2026)
- [Augmenting Interviewer Judgments of Patient Experience with Automatic Language Analysis](https://mlsanity.com/paper/2608.31007/) (2026)

## Abstract (arXiv)

Rapport, the close and harmonious relationship in which interaction partners are "in sync" with each other, was shown to result in smoother social interactions, improved collaboration, and improved interpersonal outcomes. In this work, we are first to investigate automatic prediction of low rapport during natural interactions within small groups. This task is challenging given that rapport only manifests in subtle non-verbal signals that are, in addition, subject to influences of group dynamics as well as inter-personal idiosyncrasies. We record videos of unscripted discussions of three to four people using a multi-view camera system and microphones. We analyse a rich set of non-verbal signals for rapport detection, namely facial expressions, hand motion, gaze, speaker turns, and speech prosody. Using facial features, we can detect low rapport with an average precision of 0.7 (chance level at 0.25), while incorporating prior knowledge of participants' personalities can even achieve early prediction without a drop in performance. We further provide a detailed analysis of different feature sets and the amount of information contained in different temporal segments of the interactions.