# Model-aware 3D Eye Gaze from Weak and Few-shot Supervisions

arXiv: https://arxiv.org/abs/2311.12157 · HTML: https://mlsanity.com/paper/2311.12157/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Eye Tracking for Everyone](https://mlsanity.com/paper/1606.05814/) (2016)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation](https://mlsanity.com/paper/2007.15837/) (2020)
- [It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1611.08860/) (2016)
- [Towards End-to-end Video-based Eye-Tracking](https://mlsanity.com/paper/2007.13120/) (2020)
- [TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types](https://mlsanity.com/paper/2102.02115/) (2021)
- [OpenEDS: Open Eye Dataset](https://mlsanity.com/paper/1905.03702/) (2019)
- [GazeOnce: Real-Time Multi-Person Gaze Estimation](https://mlsanity.com/paper/2204.09480/) (2022)
- [EFE: End-to-end Frame-to-Gaze Estimation](https://mlsanity.com/paper/2305.05526/) (2023)

## Built on by (most cited first, 1 of 1)

- [Hybrid-Domain Adaptative Representation Learning for Gaze Estimation](https://mlsanity.com/paper/2511.13222/) (2025)

## Abstract (arXiv)

The task of predicting 3D eye gaze from eye images can be performed either by (a) end-to-end learning for image-to-gaze mapping or by (b) fitting a 3D eye model onto images. The former case requires 3D gaze labels, while the latter requires eye semantics or landmarks to facilitate the model fitting. Although obtaining eye semantics and landmarks is relatively easy, fitting an accurate 3D eye model on them remains to be very challenging due to its ill-posed nature in general. On the other hand, obtaining large-scale 3D gaze data is cumbersome due to the required hardware setups and computational demands. In this work, we propose to predict 3D eye gaze from weak supervision of eye semantic segmentation masks and direct supervision of a few 3D gaze vectors. The proposed method combines the best of both worlds by leveraging large amounts of weak annotations--which are easy to obtain, and only a few 3D gaze vectors--which alleviate the difficulty of fitting 3D eye models on the semantic segmentation of eye images. Thus, the eye gaze vectors, used in the model fitting, are directly supervised using the few-shot gaze labels. Additionally, we propose a transformer-based network architecture, that serves as a solid baseline for our improvements. Our experiments in diverse settings illustrate the significant benefits of the proposed method, achieving about 5 degrees lower angular gaze error over the baseline, when only 0.05% 3D annotations of the training images are used. The source code is available at https://github.com/dimitris-christodoulou57/Model-aware_3D_Eye_Gaze.