# HiFiGaze: Improving Eye Tracking Accuracy Using Screen Content Knowledge

arXiv: https://arxiv.org/abs/2603.19588 · HTML: https://mlsanity.com/paper/2603.19588/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 51 references that are in the map.
- First posted on arXiv 20 Mar 2026.

## Builds on (most cited first, 17 of 17)

- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Eye Tracking for Everyone](https://mlsanity.com/paper/1606.05814/) (2016)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1611.08860/) (2016)
- [Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmark](https://mlsanity.com/paper/2104.12668/) (2021)
- [L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments](https://mlsanity.com/paper/2203.03339/) (2022)
- [WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics](https://mlsanity.com/paper/2301.13280/) (2023)
- [Adaptive Feature Fusion Network for Gaze Tracking in Mobile Tablets](https://mlsanity.com/paper/2103.11119/) (2021)
- [An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile Devices](https://mlsanity.com/paper/2307.00122/) (2023)
- [A Generalized and Robust Method Towards Practical Gaze Estimation on Smart Phone](https://mlsanity.com/paper/1910.07331/) (2019)
- [Quantifying the Impact of Motion on 2D Gaze Estimation in Real-World Mobile Interactions](https://mlsanity.com/paper/2502.10570/) (2025)
- [FreeGaze: Resource-efficient Gaze Estimation via Frequency Domain Contrastive Learning](https://mlsanity.com/paper/2209.06692/) (2022)
- [DynamicRead: Exploring Robust Gaze Interaction Methods for Reading on Handheld Mobile Devices under Dynamic Conditions](https://mlsanity.com/paper/2304.09584/) (2023)
- [Seeing the World through Your Eyes](https://mlsanity.com/paper/2306.09348/) (2023)
- [Measuring eye-tracking accuracy and its impact on usability in apple vision pro](https://mlsanity.com/paper/2406.00255/) (2024)
- [Towards Around-Device Interaction using Corneal Imaging](https://mlsanity.com/paper/1709.00966/) (2017)
- [Smartphone-based Eye Tracking System using Edge Intelligence and Model Optimisation](https://mlsanity.com/paper/2408.12463/) (2024)

## Abstract (arXiv)

We present a new and accurate approach for gaze estimation on consumer computing devices. We take advantage of continued strides in the quality of user-facing cameras found in e.g., smartphones, laptops, and desktops - 4K or greater in high-end devices - such that it is now possible to capture the 2D reflection of a device's screen in the user's eyes. This alone is insufficient for accurate gaze tracking due to the near-infinite variety of screen content. Crucially, however, the device knows what is being displayed on its own screen - in this work, we show this information allows for robust segmentation of the reflection, the location and size of which encodes the user's screen-relative gaze target. We explore several strategies to leverage this useful signal, quantifying performance in a user study. Our best performing model reduces mean tracking error by ~8% compared to a baseline appearance-based model. A supplemental study reveals an additional 10-20% improvement if the gaze-tracking camera is located at the bottom of the device.