# LAEO-Net++: revisiting people Looking At Each Other in videos

arXiv: https://arxiv.org/abs/2101.02136 · HTML: https://mlsanity.com/paper/2101.02136/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 54 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 20 of 20)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [Fine-Grained Head Pose Estimation Without Keypoints](https://mlsanity.com/paper/1710.00925/) (2017)
- [Eye Tracking for Everyone](https://mlsanity.com/paper/1606.05814/) (2016)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1711.09017/) (2017)
- [Action Tubelet Detector for Spatio-Temporal Action Localization](https://mlsanity.com/paper/1705.01861/) (2017)
- [Learnable PINs: Cross-Modal Embeddings for Person Identity](https://mlsanity.com/paper/1805.00833/) (2018)
- [Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency](https://mlsanity.com/paper/1807.10437/) (2018)
- [Self-supervised learning of a facial attribute embedding from video](https://mlsanity.com/paper/1808.06882/) (2018)
- [LAEO-Net: revisiting people Looking At Each Other in videos](https://mlsanity.com/paper/1906.05261/) (2019)
- [SALSA: A Novel Dataset for Multimodal Group Behavior Analysis](https://mlsanity.com/paper/1506.06882/) (2015)
- [Learning Interactions and Relationships between Movie Characters](https://mlsanity.com/paper/2003.13158/) (2020)
- [Tracking Gaze and Visual Focus of Attention of People Involved in Social Interaction](https://mlsanity.com/paper/1703.04727/) (2017)
- [Boosting Image-based Mutual Gaze Detection using Pseudo 3D Gaze](https://mlsanity.com/paper/2010.07811/) (2020)
- [Robust Head-Pose Estimation Based on Partially-Latent Mixture of Linear Regressions](https://mlsanity.com/paper/1603.09732/) (2016)

## Abstract (arXiv)

Capturing the 'mutual gaze' of people is essential for understanding and interpreting the social interactions between them. To this end, this paper addresses the problem of detecting people Looking At Each Other (LAEO) in video sequences. For this purpose, we propose LAEO-Net++, a new deep CNN for determining LAEO in videos. In contrast to previous works, LAEO-Net++ takes spatio-temporal tracks as input and reasons about the whole track. It consists of three branches, one for each character's tracked head and one for their relative position. Moreover, we introduce two new LAEO datasets: UCO-LAEO and AVA-LAEO. A thorough experimental evaluation demonstrates the ability of LAEO-Net++ to successfully determine if two people are LAEO and the temporal window where it happens. Our model achieves state-of-the-art results on the existing TVHID-LAEO video dataset, significantly outperforming previous approaches. Finally, we apply LAEO-Net++ to a social network, where we automatically infer the social relationship between pairs of people based on the frequency and duration that they LAEO, and show that LAEO can be a useful tool for guided search of human interactions in videos. The code is available at https://github.com/AVAuco/laeonetplus.