# Realtime Dynamic Gaze Target Tracking and Depth-Level Estimation

arXiv: https://arxiv.org/abs/2406.18595 · HTML: https://mlsanity.com/paper/2406.18595/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 53 references that are in the map.
- First posted on arXiv 9 Jun 2024.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming](https://mlsanity.com/paper/2201.08484/) (2022)
- [Pedestrian Models for Autonomous Driving Part I: Low-Level Models, from Sensing to Tracking](https://mlsanity.com/paper/2002.11669/) (2020)
- [Coordinated Control of UAVs for Human-Centered Active Sensing of Wildfires](https://mlsanity.com/paper/2006.07969/) (2020)
- [FireCommander: An Interactive, Probabilistic Multi-agent Environment for Heterogeneous Robot Teams](https://mlsanity.com/paper/2011.00165/) (2020)
- [Multi-UAV Planning for Cooperative Wildfire Coverage and Tracking with Quality-of-Service Guarantees](https://mlsanity.com/paper/2206.10544/) (2022)
- [Hardware-aware Pruning of DNNs using LFSR-Generated Pseudo-Random Indices](https://mlsanity.com/paper/1911.04468/) (2019)

## Abstract (arXiv)

The integration of Transparent Displays (TD) in various applications, such as Heads-Up Displays (HUDs) in vehicles, is a burgeoning field, poised to revolutionize user experiences. However, this innovation brings forth significant challenges in realtime human-device interaction, particularly in accurately identifying and tracking a user's gaze on dynamically changing TDs. In this paper, we present a two-fold robust and efficient systematic solution for realtime gaze monitoring, comprised of: (1) a tree-based algorithm for identifying and dynamically tracking gaze targets (i.e., moving, size-changing, and overlapping 2D content) projected on a transparent display, in realtime; (2) a multi-stream self-attention architecture to estimate the depth-level of human gaze from eye tracking data, to account for the display's transparency and preventing undesired interactions with the TD. We collected a real-world eye-tracking dataset to train and test our gaze monitoring system. We present extensive results and ablation studies, including inference experiments on System on Chip (SoC) evaluation boards, demonstrating our model's scalability, precision, and realtime feasibility in both static and dynamic contexts. Our solution marks a significant stride in enhancing next-generation user-device interaction and experience, setting a new benchmark for algorithmic gaze monitoring technology in dynamic transparent displays.