# Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels

arXiv: https://arxiv.org/abs/2502.20249 · HTML: https://mlsanity.com/paper/2502.20249/

## Key facts

- Cited by 9 papers in the map (data through 7 Oct 2026).
- Builds on 42 of its 65 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 30 of 42)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://mlsanity.com/paper/2306.00989/) (2023)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [Eye Tracking for Everyone](https://mlsanity.com/paper/1606.05814/) (2016)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation](https://mlsanity.com/paper/2007.15837/) (2020)
- [It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1611.08860/) (2016)
- [OmniMAE: Single Model Masked Pretraining on Images and Videos](https://mlsanity.com/paper/2206.08356/) (2022)
- [Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmark](https://mlsanity.com/paper/2104.12668/) (2021)
- [Detecting Attended Visual Targets in Video](https://mlsanity.com/paper/2003.02501/) (2020)
- [Gaze Estimation using Transformer](https://mlsanity.com/paper/2105.14424/) (2021)
- [A Coarse-to-Fine Adaptive Network for Appearance-Based Gaze Estimation](https://mlsanity.com/paper/2001.00187/) (2020)
- [L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments](https://mlsanity.com/paper/2203.03339/) (2022)
- [PureGaze: Purifying Gaze Feature for Generalizable Gaze Estimation](https://mlsanity.com/paper/2103.13173/) (2021)
- [Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency](https://mlsanity.com/paper/1807.10437/) (2018)
- [Appearance-Based Gaze Estimation Using Dilated-Convolutions](https://mlsanity.com/paper/1903.07296/) (2019)
- [Audio- and Gaze-driven Facial Animation of Codec Avatars](https://mlsanity.com/paper/2008.05023/) (2020)
- [Improving Few-Shot User-Specific Gaze Adaptation via Gaze Redirection Synthesis](https://mlsanity.com/paper/1904.10638/) (2019)
- [Unsupervised Representation Learning for Gaze Estimation](https://mlsanity.com/paper/1911.06939/) (2019)
- [Generalizing Gaze Estimation with Outlier-guided Collaborative Adaptation](https://mlsanity.com/paper/2107.13780/) (2021)
- [Towards End-to-end Video-based Eye-Tracking](https://mlsanity.com/paper/2007.13120/) (2020)

## Built on by (most cited first, 9 of 9)

- [OmniGaze: Reward-inspired Generalizable Gaze Estimation In The Wild](https://mlsanity.com/paper/2510.13660/) (2025)
- [Gaze Estimation for Human-Robot Interaction: Analysis Using the NICO Platform](https://mlsanity.com/paper/2509.24001/) (2025)
- [EmbedTalk: Talking Head Synthesis using Gaussian Embeddings](https://mlsanity.com/paper/2603.07604/) (2026)
- [Gaze4HRI: Zero-shot Benchmarking Gaze Estimation Neural-Networks for Human-Robot Interaction](https://mlsanity.com/paper/2605.04770/) (2026)
- [LISA: Language-guided Interference-aware Spatial-Frequency Attention for Driver Gaze Estimation](https://mlsanity.com/paper/2605.17287/) (2026)
- [GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction](https://mlsanity.com/paper/2605.22359/) (2026)
- [Semi-Supervised Gaze Estimation via Disentangled Subspace Contrastive Learning](https://mlsanity.com/paper/2605.27080/) (2026)
- [From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation](https://mlsanity.com/paper/2609.06208/) (2026)
- [Self-Supervised Multi-View 3D Gaze Target Estimation via Probabilistic Ray Marching](https://mlsanity.com/paper/2609.07415/) (2026)

## Abstract (arXiv)

Accurate 3D gaze estimation in unconstrained real-world environments remains a significant challenge due to variations in appearance, head pose, occlusion, and the limited availability of in-the-wild 3D gaze datasets. To address these challenges, we introduce a novel Self-Training Weakly-Supervised Gaze Estimation framework (ST-WSGE). This two-stage learning framework leverages diverse 2D gaze datasets, such as gaze-following data, which offer rich variations in appearances, natural scenes, and gaze distributions, and proposes an approach to generate 3D pseudo-labels and enhance model generalization. Furthermore, traditional modality-specific models, designed separately for images or videos, limit the effective use of available training data. To overcome this, we propose the Gaze Transformer (GaT), a modality-agnostic architecture capable of simultaneously learning static and dynamic gaze information from both image and video datasets. By combining 3D video datasets with 2D gaze target labels from gaze following tasks, our approach achieves the following key contributions: (i) Significant state-of-the-art improvements in within-domain and cross-domain generalization on unconstrained benchmarks like Gaze360 and GFIE, with notable cross-modal gains in video gaze estimation; (ii) Superior cross-domain performance on datasets such as MPIIFaceGaze and Gaze360 compared to frontal face methods. Code and pre-trained models will be released to the community.