# Weakly-Supervised Physically Unconstrained Gaze Estimation

arXiv: https://arxiv.org/abs/2105.09803 · HTML: https://mlsanity.com/paper/2105.09803/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 20 May 2021.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [Eye Tracking for Everyone](https://mlsanity.com/paper/1606.05814/) (2016)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation](https://mlsanity.com/paper/2007.15837/) (2020)
- [It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1611.08860/) (2016)
- [Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints](https://mlsanity.com/paper/2003.09282/) (2020)
- [Few-Shot Adaptive Gaze Estimation](https://mlsanity.com/paper/1905.01941/) (2019)
- [Detecting Attended Visual Targets in Video](https://mlsanity.com/paper/2003.02501/) (2020)
- [Learning Monocular 3D Human Pose Estimation from Multi-view Images](https://mlsanity.com/paper/1803.04775/) (2018)
- [Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild](https://mlsanity.com/paper/2003.07581/) (2020)
- [Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency](https://mlsanity.com/paper/1807.10437/) (2018)
- [Deep Pictorial Gaze Estimation](https://mlsanity.com/paper/1807.10002/) (2018)
- [Unsupervised Representation Learning for Gaze Estimation](https://mlsanity.com/paper/1911.06939/) (2019)
- [Towards End-to-end Video-based Eye-Tracking](https://mlsanity.com/paper/2007.13120/) (2020)
- [Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings](https://mlsanity.com/paper/1805.04771/) (2018)
- [LAEO-Net: revisiting people Looking At Each Other in videos](https://mlsanity.com/paper/1906.05261/) (2019)
- [A Differential Approach for Gaze Estimation](https://mlsanity.com/paper/1904.09459/) (2019)
- [Light-weight Head Pose Invariant Gaze Tracking](https://mlsanity.com/paper/1804.08572/) (2018)
- [Offset Calibration for Appearance-Based Gaze Estimation via Gaze Decomposition](https://mlsanity.com/paper/1905.04451/) (2019)

## Built on by (most cited first, 28 of 28)

- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [End-to-end Video Gaze Estimation via Capturing Head-face-eye Spatial-temporal Interaction Context](https://mlsanity.com/paper/2310.18131/) (2023)
- [An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile Devices](https://mlsanity.com/paper/2307.00122/) (2023)
- [3DGazeNet: Generalizing Gaze Estimation with Weak-Supervision from Synthetic Views](https://mlsanity.com/paper/2212.02997/) (2022)
- [EFE: End-to-end Frame-to-Gaze Estimation](https://mlsanity.com/paper/2305.05526/) (2023)
- [MTGLS: Multi-Task Gaze Estimation with Limited Supervision](https://mlsanity.com/paper/2110.12100/) (2021)
- [Free-HeadGAN: Neural Talking Head Synthesis with Explicit Gaze Control](https://mlsanity.com/paper/2208.02210/) (2022)
- [Rotation-Constrained Cross-View Feature Fusion for Multi-View Appearance-based Gaze Estimation](https://mlsanity.com/paper/2305.12704/) (2023)
- [Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels](https://mlsanity.com/paper/2502.20249/) (2025)
- [Disguise without Disruption: Utility-Preserving Face De-Identification](https://mlsanity.com/paper/2303.13269/) (2023)
- [UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Training](https://mlsanity.com/paper/2502.02307/) (2025)
- [Towards Self-Supervised Gaze Estimation](https://mlsanity.com/paper/2203.10974/) (2022)
- [Interaction-aware Joint Attention Estimation Using People Attributes](https://mlsanity.com/paper/2308.05382/) (2023)
- [Gaze-guided Hand-Object Interaction Synthesis: Dataset and Method](https://mlsanity.com/paper/2403.16169/) (2024)
- [HAZE-Net: High-Frequency Attentive Super-Resolved Gaze Estimation in Low-Resolution Face Images](https://mlsanity.com/paper/2209.10167/) (2022)
- [PrivateGaze: Preserving User Privacy in Black-box Mobile Gaze Tracking Services](https://mlsanity.com/paper/2408.00950/) (2024)
- [OmniGaze: Reward-inspired Generalizable Gaze Estimation In The Wild](https://mlsanity.com/paper/2510.13660/) (2025)
- [Modern Augmented Reality: Applications, Trends, and Future Directions](https://mlsanity.com/paper/2202.09450/) (2022)
- [AV-Gaze: A Study on the Effectiveness of Audio Guided Visual Attention Estimation for Non-Profilic Faces](https://mlsanity.com/paper/2207.03048/) (2022)
- [Investigation of Architectures and Receptive Fields for Appearance-based Gaze Estimation](https://mlsanity.com/paper/2308.09593/) (2023)
- [Learning Unsupervised Gaze Representation via Eye Mask Driven Information Bottleneck](https://mlsanity.com/paper/2407.00315/) (2024)
- [PupilSense: A Novel Application for Webcam-Based Pupil Diameter Estimation](https://mlsanity.com/paper/2407.11204/) (2024)
- [Gaze Estimation for Human-Robot Interaction: Analysis Using the NICO Platform](https://mlsanity.com/paper/2509.24001/) (2025)
- ['Labelling the Gaps': A Weakly Supervised Automatic Eye Gaze Estimation](https://mlsanity.com/paper/2208.01840/) (2022)
- [RAZE: Region Guided Self-Supervised Gaze Representation Learning](https://mlsanity.com/paper/2208.02485/) (2022)
- [Stable Tracking of Eye Gaze Direction During Ophthalmic Surgery](https://mlsanity.com/paper/2507.00635/) (2025)
- [Semi-Supervised Gaze Estimation via Disentangled Subspace Contrastive Learning](https://mlsanity.com/paper/2605.27080/) (2026)
- [Self-Supervised Multi-View 3D Gaze Target Estimation via Probabilistic Ray Marching](https://mlsanity.com/paper/2609.07415/) (2026)

## Abstract (arXiv)

A major challenge for physically unconstrained gaze estimation is acquiring training data with 3D gaze annotations for in-the-wild and outdoor scenarios. In contrast, videos of human interactions in unconstrained environments are abundantly available and can be much more easily annotated with frame-level activity labels. In this work, we tackle the previously unexplored problem of weakly-supervised gaze estimation from videos of human interactions. We leverage the insight that strong gaze-related geometric constraints exist when people perform the activity of "looking at each other" (LAEO). To acquire viable 3D gaze supervision from LAEO labels, we propose a training algorithm along with several novel loss functions especially designed for the task. With weak supervision from two large scale CMU-Panoptic and AVA-LAEO activity datasets, we show significant improvements in (a) the accuracy of semi-supervised gaze estimation and (b) cross-domain generalization on the state-of-the-art physically unconstrained in-the-wild Gaze360 gaze estimation benchmark. We open source our code at https://github.com/NVlabs/weakly-supervised-gaze.