# LAEO-Net: revisiting people Looking At Each Other in videos

arXiv: https://arxiv.org/abs/1906.05261 · HTML: https://mlsanity.com/paper/1906.05261/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2019.

## Built on by (most cited first, 30 of 34)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmark](https://mlsanity.com/paper/2104.12668/) (2021)
- [Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders](https://mlsanity.com/paper/2412.09586/) (2024)
- [ChildPlay: A New Benchmark for Understanding Children's Gaze Behaviour](https://mlsanity.com/paper/2307.01630/) (2023)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings](https://mlsanity.com/paper/2307.05158/) (2023)
- [The One Where They Reconstructed 3D Humans and Environments in TV Shows](https://mlsanity.com/paper/2207.14279/) (2022)
- [Weakly-Supervised Physically Unconstrained Gaze Estimation](https://mlsanity.com/paper/2105.09803/) (2021)
- [UniCon: Unified Context Network for Robust Active Speaker Detection](https://mlsanity.com/paper/2108.02607/) (2021)
- [Face, Body, Voice: Video Person-Clustering with Multiple Modalities](https://mlsanity.com/paper/2105.09939/) (2021)
- [Multimodal Across Domains Gaze Target Detection](https://mlsanity.com/paper/2208.10822/) (2022)
- [Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation](https://mlsanity.com/paper/2504.01020/) (2025)
- [Towards Online Multi-Modal Social Interaction Understanding](https://mlsanity.com/paper/2503.19851/) (2025)
- [3DGazeNet: Generalizing Gaze Estimation with Weak-Supervision from Synthetic Views](https://mlsanity.com/paper/2212.02997/) (2022)
- [HHP-Net: A light Heteroscedastic neural network for Head Pose estimation with uncertainty](https://mlsanity.com/paper/2111.01440/) (2021)
- [Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels](https://mlsanity.com/paper/2502.20249/) (2025)
- [How you feelin'? Learning Emotions and Mental States in Movie Scenes](https://mlsanity.com/paper/2304.05634/) (2023)
- [HAZE-Net: High-Frequency Attentive Super-Resolved Gaze Estimation in Low-Resolution Face Images](https://mlsanity.com/paper/2209.10167/) (2022)
- [MGTR: End-to-End Mutual Gaze Detection with Transformer](https://mlsanity.com/paper/2209.10930/) (2022)
- [Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features](https://mlsanity.com/paper/2312.05265/) (2023)
- [A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction](https://mlsanity.com/paper/2403.10511/) (2024)
- [Video-based Analysis Reveals Atypical Social Gaze in People with Autism Spectrum Disorder](https://mlsanity.com/paper/2409.00664/) (2024)
- [Learning Video-independent Eye Contact Segmentation from In-the-Wild Videos](https://mlsanity.com/paper/2210.02033/) (2022)
- [Sharingan: A Transformer-based Architecture for Gaze Following](https://mlsanity.com/paper/2310.00816/) (2023)
- [LAEO-Net++: revisiting people Looking At Each Other in videos](https://mlsanity.com/paper/2101.02136/) (2021)
- [Machine Learning for Brain Disorders: Transformers and Visual Transformers](https://mlsanity.com/paper/2303.12068/) (2023)
- [Stable Tracking of Eye Gaze Direction During Ophthalmic Surgery](https://mlsanity.com/paper/2507.00635/) (2025)
- [CSGaze: Context-aware Social Gaze Prediction](https://mlsanity.com/paper/2511.05955/) (2025)
- [End-to-End Shared Attention Estimation via Group Detection with Feedback Refinement](https://mlsanity.com/paper/2604.01714/) (2026)
- [Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models](https://mlsanity.com/paper/2605.19859/) (2026)

## Abstract (arXiv)

Capturing the `mutual gaze' of people is essential for understanding and interpreting the social interactions between them. To this end, this paper addresses the problem of detecting people Looking At Each Other (LAEO) in video sequences. For this purpose, we propose LAEO-Net, a new deep CNN for determining LAEO in videos. In contrast to previous works, LAEO-Net takes spatio-temporal tracks as input and reasons about the whole track. It consists of three branches, one for each character's tracked head and one for their relative position. Moreover, we introduce two new LAEO datasets: UCO-LAEO and AVA-LAEO. A thorough experimental evaluation demonstrates the ability of LAEONet to successfully determine if two people are LAEO and the temporal window where it happens. Our model achieves state-of-the-art results on the existing TVHID-LAEO video dataset, significantly outperforming previous approaches. Finally, we apply LAEO-Net to social network analysis, where we automatically infer the social relationship between pairs of people based on the frequency and duration that they LAEO.