# With Whom Do I Interact? Detecting Social Interactions in Egocentric Photo-streams

arXiv: https://arxiv.org/abs/1605.04129 · HTML: https://mlsanity.com/paper/1605.04129/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2016.

## Built on by (most cited first, 9 of 9)

- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [Visual Object Tracking in First Person Vision](https://mlsanity.com/paper/2209.13502/) (2022)
- [Is First Person Vision Challenging for Object Tracking?](https://mlsanity.com/paper/2108.13665/) (2021)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey](https://mlsanity.com/paper/2207.10574/) (2022)
- [Detecting socially interacting groups using f-formation: A survey of taxonomy, methods, datasets, applications, challenges, and future research directions](https://mlsanity.com/paper/2108.06181/) (2021)
- [Human-to-Human Interaction Detection](https://mlsanity.com/paper/2307.00464/) (2023)
- [EgoSocial: Benchmarking Proactive Intervention Ability of Omnimodal LLMs via Egocentric Social Interaction Perception](https://mlsanity.com/paper/2510.13105/) (2025)
- [Interaction-GCN: A Graph Convolutional Network based framework for social interaction recognition in egocentric videos](https://mlsanity.com/paper/2104.14007/) (2021)

## Abstract (arXiv)

Given a user wearing a low frame rate wearable camera during a day, this work aims to automatically detect the moments when the user gets engaged into a social interaction solely by reviewing the automatically captured photos by the worn camera. The proposed method, inspired by the sociological concept of F-formation, exploits distance and orientation of the appearing individuals -with respect to the user- in the scene from a bird-view perspective. As a result, the interaction pattern over the sequence can be understood as a two-dimensional time series that corresponds to the temporal evolution of the distance and orientation features over time. A Long-Short Term Memory-based Recurrent Neural Network is then trained to classify each time series. Experimental evaluation over a dataset of 30.000 images has shown promising results on the proposed method for social interaction detection in egocentric photo-streams.