mlsanityData to 7 Oct 2026

In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation

arXiv 2208.04464 ↗First posted 8 Aug 2022cs.CVcited by 51

Not on the map. Why?

Abstract (arXiv)

In this paper, we present the first transformer-based model to address the challenging problem of egocentric gaze estimation. We observe that the connection between the global scene context and local visual information is vital for localizing the gaze fixation from egocentric video frames. To this end, we design the transformer encoder to embed the global context as one additional visual token and further propose a novel Global-Local Correlation (GLC) module to explicitly model the correlation of the global token and each local token. We validate our model on two egocentric video datasets - EGTEA Gaze+ and Ego4D. Our detailed ablation studies demonstrate the benefits of our method. In addition, our approach exceeds previous state-of-the-arts by a large margin. We also provide additional visualizations to support our claim that global-local correlation serves a key representation for predicting gaze fixation from egocentric videos. More details can be found in our website (https://bolinlai.github.io/GLC-EgoGazeEst).

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 54 of its 91 references that are in the map. First posted on arXiv 8 Aug 2022.

Builds on

54 of 91 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2018SlowFast Networks for Video Recognitioncited by 1,862
  12. 2021ViViT: A Video Vision Transformercited by 1,610
38 more
  1. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  4. 2021Video Swin Transformercited by 1,090
  5. 2021Multiscale Vision Transformerscited by 896
  6. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  7. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  8. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  9. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  10. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  11. 2021Video Transformer Networkcited by 271
  12. 2021Visual Saliency Transformercited by 175
  13. 2019Gaze360: Physically Unconstrained Gaze Estimation in the Wildcited by 173
  14. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  15. 2016Eye Tracking for Everyonecited by 155
  16. 2021Salient Object Detection via Integrity Learningcited by 133
  17. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  18. 2021Shunted Self-Attention via Multi-Scale Token Aggregationcited by 90
  19. 2022TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentationcited by 82
  20. 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
  21. 2020Detecting Attended Visual Targets in Videocited by 76
  22. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  23. 2020In the Eye of the Beholder: Gaze and Actions in First Person Videocited by 68
  24. 2017SalGAN: Visual Saliency Prediction with Generative Adversarial Networkscited by 63
  25. 2019Mutual Context Network for Jointly Estimating Egocentric Gaze and Actionscited by 54
  26. 2018Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliencycited by 53
  27. 2021GLiT: Neural Architecture Search for Global and Local Image Transformercited by 53
  28. 2018EML-NET:An Expandable Multi-Layer NETwork for Saliency Predictioncited by 51
  29. 2017Video Salient Object Detection via Fully Convolutional Networkscited by 46
  30. 2019Contextual Encoder-Decoder Network for Visual Saliency Predictioncited by 44
  31. 20204D Human Body Capture from Egocentric Video via 3D Scene Groundingcited by 43
  32. 2020STAViS: Spatio-Temporal AudioVisual Saliency Networkcited by 40
  33. 2020Symbiotic Attention with Privileged Information for Egocentric Action Recognitioncited by 27
  34. 2019How is Gaze Influenced by Image Transformations? Dataset and Modelcited by 24
  35. 2020Hierarchical Domain-Adapted Feature Learning for Video Saliency Predictioncited by 24
  36. 2019A Dilated Inception Network for Visual Saliency Predictioncited by 22
  37. 2021Spatio-Temporal Self-Attention Network for Video Saliency Predictioncited by 19
  38. 2022Group Contextualization for Video Recognitioncited by 18

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 9; 2025: 21; 2026: 14.

51 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  2. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  3. 2023An Outlook into the Future of Egocentric Visioncited by 84
  4. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  5. 2022MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domaincited by 36
  6. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  7. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
  8. 2025In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Promptingcited by 21
  9. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  10. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  11. 2023Egocentric Auditory Attention Localization in Conversationscited by 18
  12. 2024Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representationscited by 18
38 more
  1. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 17
  2. 2024Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activitiescited by 15
  3. 2023The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspectivecited by 14
  4. 2025egoPPG: Heart Rate Estimation from Eye-Tracking Cameras in Egocentric Systems to Benefit Downstream Vision Taskscited by 11
  5. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  6. 2024Human Action Anticipation: A Surveycited by 8
  7. 2025EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real Worldcited by 7
  8. 2026Gaze-Regularized Vision-Language-Action Models for Robotic Manipulationcited by 7
  9. 2026Ego-Grounding for Personalized Question-Answering in Egocentric Videoscited by 6
  10. 2024Spherical World-Locking for Audio-Visual Localization in Egocentric Videoscited by 5
  11. 2024Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraintscited by 5
  12. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  13. 2025EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perceptioncited by 4
  14. 2025Uni-Hand: Universal Hand Motion Forecasting in Egocentric Viewscited by 4
  15. 2025SkillSight: Efficient First-Person Skill Assessment with Gazecited by 4
  16. 2026Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsingcited by 4
  17. 2025GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encodingcited by 3
  18. 2025Learning Predictive Visuomotor Coordinationcited by 2
  19. 2025Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Spancited by 2
  20. 2025Human Gaze Boosts Object-Centered Representation Learningcited by 1
  21. 2025Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reachcited by 1
  22. 2026ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimationcited by 1
  23. 2025Personalized Federated Learning for Egocentric Video Gaze Estimation with Comprehensive Parameter Frezzingcited by 0
  24. 2025Neural Radiance and Gaze Fields for Visual Attention Modeling in 3D Environmentscited by 0
  25. 2025Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalitiescited by 0
  26. 2025GazeNLQ @ Ego4D Natural Language Queries Challenge 2025cited by 0
  27. 2025Seeing My Future: Predicting Situated Interaction Behavior in Virtual Realitycited by 0
  28. 2025HAGI++: Head-Assisted Gaze Imputation and Generationcited by 0
  29. 2025EgoCampus: Egocentric Pedestrian Eye Gaze Model and Datasetcited by 0
  30. 2026Robust Egocentric Visual Attention Prediction Through Language-guided Scene Context-aware Learningcited by 0
  31. 2026Temporal Slowness in Central Vision Drives Semantic Object Learningcited by 0
  32. 2026Egocentric Gaze Estimation via Neck-Mounted Cameracited by 0
  33. 2026Real Eyes Realize Faster: Gaze Stability and Pupil Novelty for Efficient Egocentric Learningcited by 0
  34. 2026GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformercited by 0
  35. 2026How Much Future Helps? A Controlled Study of Future-Privileged Supervision for Causal Egocentric Gaze Estimationcited by 0
  36. 2026SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understandingcited by 0
  37. 2026HiResNets: Native Full-HD Video Recognition with Foveal Residual Streamscited by 0
  38. 2026EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Inputcited by 0

50 most cited of 51 All 51 (JSON)