# Gaze Estimation using Transformer

arXiv: https://arxiv.org/abs/2105.14424 · HTML: https://mlsanity.com/paper/2105.14424/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 30 May 2021.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation](https://mlsanity.com/paper/2007.15837/) (2020)
- [It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1611.08860/) (2016)
- [MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1711.09017/) (2017)
- [Colorization Transformer](https://mlsanity.com/paper/2102.04432/) (2021)
- [Scalable Vision Transformers with Hierarchical Pooling](https://mlsanity.com/paper/2103.10619/) (2021)
- [Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmark](https://mlsanity.com/paper/2104.12668/) (2021)
- [Deep Visual Attention Prediction](https://mlsanity.com/paper/1705.02544/) (2017)
- [A Coarse-to-Fine Adaptive Network for Appearance-Based Gaze Estimation](https://mlsanity.com/paper/2001.00187/) (2020)
- [PureGaze: Purifying Gaze Feature for Generalizable Gaze Estimation](https://mlsanity.com/paper/2103.13173/) (2021)
- [Deep Pictorial Gaze Estimation](https://mlsanity.com/paper/1807.10002/) (2018)
- [Appearance-Based Gaze Estimation Using Dilated-Convolutions](https://mlsanity.com/paper/1903.07296/) (2019)
- [RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decoder](https://mlsanity.com/paper/2010.15831/) (2020)

## Built on by (most cited first, 30 of 68)

- [Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions](https://mlsanity.com/paper/2404.07214/) (2024)
- [A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming](https://mlsanity.com/paper/2404.16038/) (2024)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [GazeOnce: Real-Time Multi-Person Gaze Estimation](https://mlsanity.com/paper/2204.09480/) (2022)
- [TinyTracker: Ultra-Fast and Ultra-Low-Power Edge Vision In-Sensor for Gaze Estimation](https://mlsanity.com/paper/2307.07813/) (2023)
- [DVGaze: Dual-View Gaze Estimation](https://mlsanity.com/paper/2308.10310/) (2023)
- [End-to-end Video Gaze Estimation via Capturing Head-face-eye Spatial-temporal Interaction Context](https://mlsanity.com/paper/2310.18131/) (2023)
- [What Do You See in Vehicle? Comprehensive Vision Solution for In-Vehicle Gaze Estimation](https://mlsanity.com/paper/2403.15664/) (2024)
- [An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile Devices](https://mlsanity.com/paper/2307.00122/) (2023)
- [Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation](https://mlsanity.com/paper/2404.05215/) (2024)
- [3DGazeNet: Generalizing Gaze Estimation with Weak-Supervision from Synthetic Views](https://mlsanity.com/paper/2212.02997/) (2022)
- [Rotation-Constrained Cross-View Feature Fusion for Multi-View Appearance-based Gaze Estimation](https://mlsanity.com/paper/2305.12704/) (2023)
- [GazeGaussian: High-Fidelity Gaze Redirection with 3D Gaussian Splatting](https://mlsanity.com/paper/2411.12981/) (2024)
- [Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels](https://mlsanity.com/paper/2502.20249/) (2025)
- [UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Training](https://mlsanity.com/paper/2502.02307/) (2025)
- [High-Fidelity Eye Animatable Neural Radiance Fields for Human Face](https://mlsanity.com/paper/2308.00773/) (2023)
- [Multi-Modal Gaze Following in Conversational Scenarios](https://mlsanity.com/paper/2311.05669/) (2023)
- [TransGOP: Transformer-Based Gaze Object Prediction](https://mlsanity.com/paper/2402.13578/) (2024)
- [Quantifying the Impact of Motion on 2D Gaze Estimation in Real-World Mobile Interactions](https://mlsanity.com/paper/2502.10570/) (2025)
- [OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis](https://mlsanity.com/paper/2506.02891/) (2025)
- [FLAME: Facial Landmark Heatmap Activated Multimodal Gaze Estimation](https://mlsanity.com/paper/2110.04828/) (2021)
- [Modeling State Shifting via Local-Global Distillation for Event-Frame Gaze Tracking](https://mlsanity.com/paper/2404.00548/) (2024)
- [OmniGaze: Reward-inspired Generalizable Gaze Estimation In The Wild](https://mlsanity.com/paper/2510.13660/) (2025)
- [Toward Super-Resolution for Appearance-Based Gaze Estimation](https://mlsanity.com/paper/2303.10151/) (2023)
- [FR-Net:A Light-weight FFT Residual Net For Gaze Estimation](https://mlsanity.com/paper/2305.11875/) (2023)
- [Domain-Adaptive Full-Face Gaze Estimation via Novel-View-Synthesis and Feature Disentanglement](https://mlsanity.com/paper/2305.16140/) (2023)
- [3D Prior is All You Need: Cross-Task Few-shot 2D Gaze Estimation](https://mlsanity.com/paper/2502.04074/) (2025)
- [Differential Contrastive Training for Gaze Estimation](https://mlsanity.com/paper/2502.20128/) (2025)
- [MAC-Gaze: Motion-Aware Continual Calibration for Mobile Gaze Tracking](https://mlsanity.com/paper/2505.22769/) (2025)
- [Learning in Focus: Detecting Behavioral and Collaborative Engagement Using Vision Transformers](https://mlsanity.com/paper/2508.15782/) (2025)

## Abstract (arXiv)

Recent work has proven the effectiveness of transformers in many computer vision tasks. However, the performance of transformers in gaze estimation is still unexplored. In this paper, we employ transformers and assess their effectiveness for gaze estimation. We consider two forms of vision transformer which are pure transformers and hybrid transformers. We first follow the popular ViT and employ a pure transformer to estimate gaze from images. On the other hand, we preserve the convolutional layers and integrate CNNs as well as transformers. The transformer serves as a component to complement CNNs. We compare the performance of the two transformers in gaze estimation. The Hybrid transformer significantly outperforms the pure transformer in all evaluation datasets with less parameters. We further conduct experiments to assess the effectiveness of the hybrid transformer and explore the advantage of self-attention mechanism. Experiments show the hybrid transformer can achieve state-of-the-art performance in all benchmarks with pre-training.To facilitate further research, we release codes and models in https://github.com/yihuacheng/GazeTR.