# Let me join you! Real-time F-formation recognition by a socially aware robot

arXiv: https://arxiv.org/abs/2008.10078 · HTML: https://mlsanity.com/paper/2008.10078/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2020.

## Built on by (most cited first, 5 of 5)

- [Real-time Trajectory-based Social Group Detection](https://mlsanity.com/paper/2304.05678/) (2023)
- [Detecting socially interacting groups using f-formation: A survey of taxonomy, methods, datasets, applications, challenges, and future research directions](https://mlsanity.com/paper/2108.06181/) (2021)
- [A Perspective on Robotic Telepresence and Teleoperation using Cognition: Are we there yet?](https://mlsanity.com/paper/2203.02959/) (2022)
- [Exploiting temporal information to detect conversational groups in videos and predict the next speaker](https://mlsanity.com/paper/2408.16380/) (2024)
- [Where Should I Join? Robot Group Joining via Language-Guided Goal Prediction](https://mlsanity.com/paper/2609.28467/) (2026)

## Abstract (arXiv)

This paper presents a novel architecture to detect social groups in real-time from a continuous image stream of an ego-vision camera. F-formation defines social orientations in space where two or more person tends to communicate in a social place. Thus, essentially, we detect F-formations in social gatherings such as meetings, discussions, etc. and predict the robot's approach angle if it wants to join the social group. Additionally, we also detect outliers, i.e., the persons who are not part of the group under consideration. Our proposed pipeline consists of -- a) a skeletal key points estimator (a total of 17) for the detected human in the scene, b) a learning model (using a feature vector based on the skeletal points) using CRF to detect groups of people and outlier person in a scene, and c) a separate learning model using a multi-class Support Vector Machine (SVM) to predict the exact F-formation of the group of people in the current scene and the angle of approach for the viewing robot. The system is evaluated using two data-sets. The results show that the group and outlier detection in a scene using our method establishes an accuracy of 91%. We have made rigorous comparisons of our systems with a state-of-the-art F-formation detection system and found that it outperforms the state-of-the-art by 29% for formation detection and 55% for combined detection of the formation and approach angle.