# FovealNet: Advancing AI-Driven Gaze Tracking Solutions for Optimized Foveated Rendering System Performance in Virtual Reality

arXiv: https://arxiv.org/abs/2412.10456 · HTML: https://mlsanity.com/paper/2412.10456/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 67 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 14 of 14)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer](https://mlsanity.com/paper/2108.01390/) (2021)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1711.09017/) (2017)
- [FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality](https://mlsanity.com/paper/2103.16365/) (2021)
- [Event Based, Near Eye Gaze Tracking Beyond 10,000Hz](https://mlsanity.com/paper/2004.03577/) (2020)
- [RITnet: Real-time Semantic Segmentation of the Eye for Gaze Tracking](https://mlsanity.com/paper/1910.00694/) (2019)
- [Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications](https://mlsanity.com/paper/1901.10906/) (2019)
- [EllSeg: An Ellipse Segmentation Framework for Robust Gaze Tracking](https://mlsanity.com/paper/2007.09600/) (2020)
- [Real-Time Gaze Tracking with Event-Driven Eye Segmentation](https://mlsanity.com/paper/2201.07367/) (2022)
- [FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks](https://mlsanity.com/paper/2209.09965/) (2022)
- [One Eye is All You Need: Lightweight Ensembles for Gaze Estimation with Single Encoders](https://mlsanity.com/paper/2211.11936/) (2022)
- [Saccade Sequence Prediction: Beyond Static Saliency Maps](https://mlsanity.com/paper/1711.10959/) (2017)

## Built on by (most cited first, 1 of 1)

- [A3FR: Agile 3D Gaussian Splatting with Incremental Gaze Tracked Foveated Rendering in Virtual Reality](https://mlsanity.com/paper/2507.04147/) (2025)

## Abstract (arXiv)

Leveraging real-time eye-tracking, foveated rendering optimizes hardware efficiency and enhances visual quality virtual reality (VR). This approach leverages eye-tracking techniques to determine where the user is looking, allowing the system to render high-resolution graphics only in the foveal region-the small area of the retina where visual acuity is highest, while the peripheral view is rendered at lower resolution. However, modern deep learning-based gaze-tracking solutions often exhibit a long-tail distribution of tracking errors, which can degrade user experience and reduce the benefits of foveated rendering by causing misalignment and decreased visual quality. This paper introduces textitFovealNet, an advanced AI-driven gaze tracking framework designed to optimize system performance by strategically enhancing gaze tracking accuracy. To further reduce the implementation cost of the gaze tracking algorithm, FovealNet employs an event-based cropping method that eliminates over 64.8% of irrelevant pixels from the input image. Additionally, it incorporates a simple yet effective token-pruning strategy that dynamically removes tokens on the fly without compromising tracking accuracy. Finally, to support different runtime rendering configurations, we propose a system performance-aware multi-resolution training strategy, allowing the gaze tracking DNN to adapt and optimize overall system performance more effectively. Evaluation results demonstrate that FovealNet achieves at least 1.42× speed up compared to previous methods and 13% increase in perceptual quality for foveated output.