# Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation

arXiv: https://arxiv.org/abs/2404.05215 · HTML: https://mlsanity.com/paper/2404.05215/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 74 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [Eye Tracking for Everyone](https://mlsanity.com/paper/1606.05814/) (2016)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1611.08860/) (2016)
- [Few-Shot Adaptive Gaze Estimation](https://mlsanity.com/paper/1905.01941/) (2019)
- [ActionVLAD: Learning spatio-temporal aggregation for action classification](https://mlsanity.com/paper/1704.02895/) (2017)
- [Gaze Estimation using Transformer](https://mlsanity.com/paper/2105.14424/) (2021)
- [L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments](https://mlsanity.com/paper/2203.03339/) (2022)
- [Pre-trained Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2109.08215/) (2021)
- [Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors](https://mlsanity.com/paper/1505.04868/) (2015)
- [Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency](https://mlsanity.com/paper/1807.10437/) (2018)
- [Deep Pictorial Gaze Estimation](https://mlsanity.com/paper/1807.10002/) (2018)
- [Describing and Localizing Multiple Changes with Transformers](https://mlsanity.com/paper/2103.14146/) (2021)
- [Integrating Human Gaze into Attention for Egocentric Activity Recognition](https://mlsanity.com/paper/2011.03920/) (2020)
- [Object Referring in Videos with Language and Human Gaze](https://mlsanity.com/paper/1801.01582/) (2018)
- [Towards End-to-end Video-based Eye-Tracking](https://mlsanity.com/paper/2007.13120/) (2020)
- [Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings](https://mlsanity.com/paper/1805.04771/) (2018)
- [Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications](https://mlsanity.com/paper/1901.10906/) (2019)
- [A Differential Approach for Gaze Estimation](https://mlsanity.com/paper/1904.09459/) (2019)
- [Recurrent CNN for 3D Gaze Estimation using Appearance and Shape Cues](https://mlsanity.com/paper/1805.03064/) (2018)

## Built on by (most cited first, 12 of 12)

- [Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels](https://mlsanity.com/paper/2502.20249/) (2025)
- [GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding](https://mlsanity.com/paper/2511.06348/) (2025)
- [3D Prior is All You Need: Cross-Task Few-shot 2D Gaze Estimation](https://mlsanity.com/paper/2502.04074/) (2025)
- [A Survey on Deep Learning-based Gaze Direction Regression: Searching for the State-of-the-art](https://mlsanity.com/paper/2410.17082/) (2024)
- [DHECA-SuperGaze: Dual Head-Eye Cross-Attention and Super-Resolution for Unconstrained Gaze Estimation](https://mlsanity.com/paper/2505.08426/) (2025)
- [MAGE: A Multi-task Architecture for Gaze Estimation with an Efficient Calibration Module](https://mlsanity.com/paper/2505.16384/) (2025)
- [CapStARE: Capsule-based Sequential Architecture for Robust and Efficient Gaze Estimation](https://mlsanity.com/paper/2509.19936/) (2025)
- [GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation](https://mlsanity.com/paper/2601.12948/) (2026)
- [PicoEyes: Unified Gaze Estimation Framework for Mixed Reality with a Large-Scale Multi-View Dataset](https://mlsanity.com/paper/2605.07188/) (2026)
- [LISA: Language-guided Interference-aware Spatial-Frequency Attention for Driver Gaze Estimation](https://mlsanity.com/paper/2605.17287/) (2026)
- [From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation](https://mlsanity.com/paper/2609.06208/) (2026)
- [EyeTAG: Eye Trajectory-Aware Gaze Estimation](https://mlsanity.com/paper/2610.00922/) (2026)

## Abstract (arXiv)

Gaze is an essential prompt for analyzing human behavior and attention. Recently, there has been an increasing interest in determining gaze direction from facial videos. However, video gaze estimation faces significant challenges, such as understanding the dynamic evolution of gaze in video sequences, dealing with static backgrounds, and adapting to variations in illumination. To address these challenges, we propose a simple and novel deep learning model designed to estimate gaze from videos, incorporating a specialized attention module. Our method employs a spatial attention mechanism that tracks spatial dynamics within videos. This technique enables accurate gaze direction prediction through a temporal sequence model, adeptly transforming spatial observations into temporal insights, thereby significantly improving gaze estimation accuracy. Additionally, our approach integrates Gaussian processes to include individual-specific traits, facilitating the personalization of our model with just a few labeled samples. Experimental results confirm the efficacy of the proposed approach, demonstrating its success in both within-dataset and cross-dataset settings. Specifically, our proposed approach achieves state-of-the-art performance on the Gaze360 dataset, improving by 2.5^circ without personalization. Further, by personalizing the model with just three samples, we achieved an additional improvement of 0.8^circ. The code and pre-trained models are available at urlhttps://github.com/jswati31/stage.