# LEyes: A Lightweight Framework for Deep Learning-Based Eye Tracking using Synthetic Eye Images

arXiv: https://arxiv.org/abs/2309.06129 · HTML: https://mlsanity.com/paper/2309.06129/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 95 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [Eye Tracking for Everyone](https://mlsanity.com/paper/1606.05814/) (2016)
- [Simulation-based reinforcement learning for real-world autonomous driving](https://mlsanity.com/paper/1911.12905/) (2019)
- [Rendering of Eyes for Eye-Shape Registration and Gaze Estimation](https://mlsanity.com/paper/1505.05916/) (2015)
- [Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings](https://mlsanity.com/paper/1805.04771/) (2018)
- [TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types](https://mlsanity.com/paper/2102.02115/) (2021)
- [Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities](https://mlsanity.com/paper/1905.13146/) (2019)
- [EllSeg: An Ellipse Segmentation Framework for Robust Gaze Tracking](https://mlsanity.com/paper/2007.09600/) (2020)
- [PupilNet: Convolutional Neural Networks for Robust Pupil Detection](https://mlsanity.com/paper/1601.04902/) (2016)
- [A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms](https://mlsanity.com/paper/1708.01817/) (2017)
- [PuRe: Robust pupil detection for real-time pervasive eye tracking](https://mlsanity.com/paper/1712.08900/) (2017)
- [Deep semantic gaze embedding and scanpath comparison for expertise classification during OPT viewing](https://mlsanity.com/paper/2003.13987/) (2020)
- [PupilNet v2.0: Convolutional Neural Networks for CPU based real time Robust Pupil Detection](https://mlsanity.com/paper/1711.00112/) (2017)
- [RIT-Eyes: Rendering of near-eye images for eye-tracking applications](https://mlsanity.com/paper/2006.03642/) (2020)
- [Quantitative Digital Microscopy with Deep Learning](https://mlsanity.com/paper/2010.08260/) (2020)
- [Pistol: Pupil Invisible Supportive Tool to extract Pupil, Iris, Eye Opening, Eye Movements, Pupil and Iris Gaze Vector, and 2D as well as 3D Gaze](https://mlsanity.com/paper/2201.06799/) (2022)
- [EllSeg-Gen, towards Domain Generalization for head-mounted eyetracking](https://mlsanity.com/paper/2205.01947/) (2022)
- [Eye Semantic Segmentation with a Lightweight Model](https://mlsanity.com/paper/1911.01049/) (2019)
- [Precise localization of corneal reflections in eye images using deep learning trained on synthetic data](https://mlsanity.com/paper/2304.05673/) (2023)

## Built on by (most cited first, 4 of 4)

- [Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)](https://mlsanity.com/paper/2311.08077/) (2023)
- [Zero-Shot Pupil Segmentation with SAM 2: A Case Study of Over 14 Million Images](https://mlsanity.com/paper/2410.08926/) (2024)
- [Eye image segmentation using visual and concept prompts with Segment Anything Model 3 (SAM3)](https://mlsanity.com/paper/2603.17715/) (2026)
- [Night Eyes: A Reproducible Framework for Constellation-Based Corneal Reflection Matching](https://mlsanity.com/paper/2604.01909/) (2026)

## Abstract (arXiv)

Deep learning has bolstered gaze estimation techniques, but real-world deployment has been impeded by inadequate training datasets. This problem is exacerbated by both hardware-induced variations in eye images and inherent biological differences across the recorded participants, leading to both feature and pixel-level variance that hinders the generalizability of models trained on specific datasets. While synthetic datasets can be a solution, their creation is both time and resource-intensive. To address this problem, we present a framework called Light Eyes or "LEyes" which, unlike conventional photorealistic methods, only models key image features required for video-based eye tracking using simple light distributions. LEyes facilitates easy configuration for training neural networks across diverse gaze-estimation tasks. We demonstrate that models trained using LEyes are consistently on-par or outperform other state-of-the-art algorithms in terms of pupil and CR localization across well-known datasets. In addition, a LEyes trained model outperforms the industry standard eye tracker using significantly more cost-effective hardware. Going forward, we are confident that LEyes will revolutionize synthetic data generation for gaze estimation models, and lead to significant improvements of the next generation video-based eye trackers.