# Digitally Prototype Your Eye Tracker: Simulating Hardware Performance using 3D Synthetic Data

arXiv: https://arxiv.org/abs/2503.16742 · HTML: https://mlsanity.com/paper/2503.16742/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Builds on (most cited first, 16 of 16)

- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [Few-Shot Adaptive Gaze Estimation](https://mlsanity.com/paper/1905.01941/) (2019)
- [Rendering of Eyes for Eye-Shape Registration and Gaze Estimation](https://mlsanity.com/paper/1505.05916/) (2015)
- [Dancing under the stars: video denoising in starlight](https://mlsanity.com/paper/2204.04210/) (2022)
- [Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction](https://mlsanity.com/paper/1405.0006/) (2014)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [ElSe: Ellipse Selection for Robust Pupil Detection in Real-World Environments](https://mlsanity.com/paper/1511.06575/) (2015)
- [EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes](https://mlsanity.com/paper/2206.08428/) (2022)
- [Aberration-Aware Depth-from-Focus](https://mlsanity.com/paper/2303.04654/) (2023)
- [RIT-Eyes: Rendering of near-eye images for eye-tracking applications](https://mlsanity.com/paper/2006.03642/) (2020)
- [Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking Systems](https://mlsanity.com/paper/2403.15947/) (2024)
- [Seeing the World through Your Eyes](https://mlsanity.com/paper/2306.09348/) (2023)
- [Lens Factory: Automatic Lens Generation Using Off-the-shelf Components](https://mlsanity.com/paper/1506.08956/) (2015)

## Built on by (most cited first, 2 of 2)

- [Rapidly deploying on-device eye tracking by distilling visual foundation models](https://mlsanity.com/paper/2604.02509/) (2026)
- [GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction](https://mlsanity.com/paper/2605.22359/) (2026)

## Abstract (arXiv)

Eye tracking (ET) is a key enabler for Augmented and Virtual Reality (AR/VR). Prototyping new ET hardware requires assessing the impact of hardware choices on eye tracking performance. This task is compounded by the high cost of obtaining data from sufficiently many variations of real hardware, especially for machine learning, which requires large training datasets. We propose a method for end-to-end evaluation of how hardware changes impact machine learning-based ET performance using only synthetic data. We utilize a dataset of real 3D eyes, reconstructed from light dome data using neural radiance fields (NeRF), to synthesize captured eyes from novel viewpoints and camera parameters. Using this framework, we demonstrate that we can predict the relative performance across various hardware configurations, accounting for variations in sensor noise, illumination brightness, and optical blur. We also compare our simulator with the publicly available eye tracking dataset from the Project Aria glasses, demonstrating a strong correlation with real-world performance. Finally, we present a first-of-its-kind analysis in which we vary ET camera positions, evaluating ET performance ranging from on-axis direct views of the eye to peripheral views on the frame. Such an analysis would have previously required manufacturing physical devices to capture evaluation data. In short, our method enables faster prototyping of ET hardware.