# End-to-end Video Gaze Estimation via Capturing Head-face-eye Spatial-temporal Interaction Context

arXiv: https://arxiv.org/abs/2310.18131 · HTML: https://mlsanity.com/paper/2310.18131/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1611.08860/) (2016)
- [Gaze Estimation using Transformer](https://mlsanity.com/paper/2105.14424/) (2021)
- [A Coarse-to-Fine Adaptive Network for Appearance-Based Gaze Estimation](https://mlsanity.com/paper/2001.00187/) (2020)
- [L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments](https://mlsanity.com/paper/2203.03339/) (2022)
- [Temporally Efficient Vision Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2204.08412/) (2022)
- [Appearance-Based Gaze Estimation Using Dilated-Convolutions](https://mlsanity.com/paper/1903.07296/) (2019)
- [Local-Global Context Aware Transformer for Language-Guided Video Segmentation](https://mlsanity.com/paper/2203.09773/) (2022)
- [Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning](https://mlsanity.com/paper/1909.02144/) (2019)
- [Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications](https://mlsanity.com/paper/1901.10906/) (2019)
- [Adaptive Feature Fusion Network for Gaze Tracking in Mobile Tablets](https://mlsanity.com/paper/2103.11119/) (2021)
- [Weakly-Supervised Physically Unconstrained Gaze Estimation](https://mlsanity.com/paper/2105.09803/) (2021)
- [Local-Global Temporal Difference Learning for Satellite Video Super-Resolution](https://mlsanity.com/paper/2304.04421/) (2023)
- [Precise Facial Landmark Detection by Reference Heatmap Transformer](https://mlsanity.com/paper/2303.07840/) (2023)
- [The Story in Your Eyes: An Individual-difference-aware Model for Cross-person Gaze Estimation](https://mlsanity.com/paper/2106.14183/) (2021)
- [Real-time Multi-person Eyeblink Detection in the Wild for Untrimmed Video](https://mlsanity.com/paper/2303.16053/) (2023)

## Built on by (most cited first, 14 of 14)

- [Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels](https://mlsanity.com/paper/2502.20249/) (2025)
- [UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Training](https://mlsanity.com/paper/2502.02307/) (2025)
- [Gaze-guided Hand-Object Interaction Synthesis: Dataset and Method](https://mlsanity.com/paper/2403.16169/) (2024)
- [OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis](https://mlsanity.com/paper/2506.02891/) (2025)
- [Appearance Debiased Gaze Estimation via Stochastic Subject-Wise Adversarial Learning](https://mlsanity.com/paper/2401.13865/) (2024)
- [Improving Domain Generalization on Gaze Estimation via Branch-out Auxiliary Regularization](https://mlsanity.com/paper/2405.01439/) (2024)
- [3D Gaze Tracking for Studying Collaborative Interactions in Mixed-Reality Environments](https://mlsanity.com/paper/2406.11003/) (2024)
- [A Survey on Deep Learning-based Gaze Direction Regression: Searching for the State-of-the-art](https://mlsanity.com/paper/2410.17082/) (2024)
- [Investigating Bias and Fairness in Appearance-based Gaze Estimation](https://mlsanity.com/paper/2604.10707/) (2026)
- [DHECA-SuperGaze: Dual Head-Eye Cross-Attention and Super-Resolution for Unconstrained Gaze Estimation](https://mlsanity.com/paper/2505.08426/) (2025)
- [GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation](https://mlsanity.com/paper/2601.12948/) (2026)
- [Gaze4HRI: Zero-shot Benchmarking Gaze Estimation Neural-Networks for Human-Robot Interaction](https://mlsanity.com/paper/2605.04770/) (2026)
- [Self-Supervised Multi-View 3D Gaze Target Estimation via Probabilistic Ray Marching](https://mlsanity.com/paper/2609.07415/) (2026)
- [EyeTAG: Eye Trajectory-Aware Gaze Estimation](https://mlsanity.com/paper/2610.00922/) (2026)

## Abstract (arXiv)

In this letter, we propose a new method, Multi-Clue Gaze (MCGaze), to facilitate video gaze estimation via capturing spatial-temporal interaction context among head, face, and eye in an end-to-end learning way, which has not been well concerned yet. The main advantage of MCGaze is that the tasks of clue localization of head, face, and eye can be solved jointly for gaze estimation in a one-step way, with joint optimization to seek optimal performance. During this, spatial-temporal context exchange happens among the clues on the head, face, and eye. Accordingly, the final gazes obtained by fusing features from various queries can be aware of global clues from heads and faces, and local clues from eyes simultaneously, which essentially leverages performance. Meanwhile, the one-step running way also ensures high running efficiency. Experiments on the challenging Gaze360 dataset verify the superiority of our proposition. The source code will be released at https://github.com/zgchen33/MCGaze.